In this project, a dataset is given with 900+ data each corresponding to individual indicators of users' experience surveys revolving around proposed system designs -- and how would potential users react or perceive this notion.
The goal is to provide a deep learning model with an acceptable rate of AUC (0.7 to 0.9) on test dataset in order to be able to predict certain words clearly.
To gain a better sense of the data, we first import the necessities to run the overall code.
# Configure output settings to prettify print display
from IPython.display import HTML, display
def set_css():
display(HTML('''
<style>
pre {
white-space: pre-wrap;
}
</style>
'''))
get_ipython().events.register('pre_run_cell', set_css)
# Ignore warnings to prettify print
import warnings
warnings.filterwarnings("ignore")
# !pip install scikit-multilearn
import pandas as pd
import tensorflow as tf
import numpy as np
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
import imblearn
print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))
Num GPUs Available: 1
We can then import the data via pandas package to see the overall nature of the data, the outputs, and the datatypes.
df = pd.read_excel('Charis-avg3.xlsx',
sheet_name='avg3-round', header=[0], engine='openpyxl')
df.drop(columns=['ID', 'ProjectID', 'Class'], axis=1, inplace=True)
df.sample(n=10, random_state=1)
| Requirements | supportive | easy | efficient | clear | exciting | interesting | inventive | leading edge | |
|---|---|---|---|---|---|---|---|---|---|
| 718 | Secure data transfer over the Internet could b... | 7 | 4 | 6 | 6 | 4 | 5 | 3 | 3 |
| 637 | The system shall validate the amount is availa... | 6 | 6 | 7 | 7 | 4 | 4 | 3 | 3 |
| 759 | Sensitive data will be encrypted before being ... | 7 | 5 | 7 | 6 | 4 | 4 | 3 | 3 |
| 366 | PME Subsystem shall allow building product men... | 6 | 5 | 6 | 6 | 4 | 4 | 5 | 5 |
| 543 | The system shall be responsive. At the DePaul ... | 7 | 5 | 7 | 7 | 6 | 6 | 4 | 4 |
| 41 | The product shall synchronize with the office ... | 5 | 5 | 5 | 5 | 4 | 4 | 4 | 4 |
| 120 | Program Administrators and Nursing Staff Membe... | 6 | 6 | 6 | 6 | 5 | 5 | 3 | 3 |
| 725 | It must be able to download the most recent me... | 5 | 6 | 6 | 6 | 4 | 5 | 4 | 3 |
| 795 | Customer must logout of the site after purchas... | 4 | 5 | 4 | 5 | 2 | 2 | 3 | 4 |
| 672 | Changes made to the Manage My ID website can b... | 5 | 6 | 6 | 5 | 5 | 5 | 5 | 5 |
Features are only represented in the first column, which is the Requirements column. This column represents the question and/or the funcionalities of the system itself.
Labels, on the other hand, comprised of 8 columns in total. They are:
The scales on the labels vary from 1 to 7, in which values closer to 1 corresponds to a more negative connotation, while values closer to 7 is likely to be more positive.
For example, for column boring<==>exciting, if a user gives it a 3, whilst this may seem neutral at first glance, we can infer that the user does not regard this system design as easy-to-use -- otherwise they would give it a score of 5 or 6.
For this task, we'll run multiclass multilabel regression since we obtained a good AUC with multiclass regression on one column before with simple regression. The trick here is to not use a single column.
With this, we might want to approach the problem within the scope of Multi-output Regression
First, we separate the labels from the data.
features = df.iloc[:, 0].values # First column in the entire set
labels = df.iloc[:, np.r_[1:9]].values
print("Features is now of type:", type(features), "with shape", features.shape)
print("Labels is now of type:", type(labels), "with shape", labels.shape)
Features is now of type: <class 'numpy.ndarray'> with shape (969,) Labels is now of type: <class 'numpy.ndarray'> with shape (969, 8)
We know from previous work that the dataset is very much imbalanced, so we would want to try oversampling across all columns and make them our new label. But first, we would want to convert features as word vectors first.
We can visualize the most used words via wordcloud used in our dataset.
%matplotlib inline
from matplotlib import pyplot as plt
from wordcloud import WordCloud,STOPWORDS
plt.figure(figsize=(30,20))
text =df['Requirements'].values
cloud = WordCloud(
stopwords=STOPWORDS,
background_color='white',
collocations=False,
width=1920,
height=1080
).generate(" ".join(text))
plt.axis('off')
plt.title("Word Cloud for our Dataset",fontsize=40)
plt.imshow(cloud)
<matplotlib.image.AxesImage at 0x7f00e0bdc450>
from collections import Counter
for i in range(8):
print("Column:", df.columns[1:][i])
print(Counter(labels[:, i]))
Column: supportive
Counter({6: 476, 7: 272, 5: 173, 4: 41, 3: 6, 2: 1})
Column: easy
Counter({6: 491, 5: 295, 7: 90, 4: 81, 3: 10, 2: 2})
Column: efficient
Counter({6: 546, 7: 251, 5: 140, 4: 28, 3: 4})
Column: clear
Counter({6: 490, 7: 242, 5: 200, 4: 32, 3: 4, 2: 1})
Column: exciting
Counter({4: 465, 5: 313, 3: 106, 6: 76, 7: 5, 2: 4})
Column: interesting
Counter({4: 406, 5: 387, 3: 96, 6: 73, 7: 5, 2: 2})
Column: inventive
Counter({4: 438, 3: 316, 5: 124, 2: 72, 6: 14, 1: 5})
Column: leading edge
Counter({4: 428, 3: 312, 5: 147, 2: 60, 6: 16, 1: 6})
Tokenization means splitting each word and vectorizing a text corpus, by turning each text into either a sequence of integers (each integer being the index of a token in a dictionary). This process cleans up our text, lowercase, and remove punctuations. This step is basically the same from our previous work involving LSTMs.
This way, we can see which words appear the most in our training set.
tokenizer = Tokenizer(num_words=2000, oov_token="<OOV>")
tokenizer.fit_on_texts(features)
word_index = tokenizer.word_index
dict(list(word_index.items())[:10])
{'<OOV>': 1,
'a': 8,
'and': 9,
'be': 7,
'of': 6,
'product': 10,
'shall': 3,
'system': 5,
'the': 2,
'to': 4}
After tokenization, the next step is to turn those tokens into lists of sequence.
sequences = tokenizer.texts_to_sequences(features)
print(sequences[0])
# This corresponds to the first sentence in the training data.
[2, 5, 3, 672, 2, 32, 95, 673, 49]
When we train neural networks for NLP, we need sequences to be in the same size, that's why we use padding. Padding means that we convert all instances of the training data towards the same size. To do this, we need to specify a fixed length to transform all the instances on.
For the project, we can set the sequence by looking quickly at the average proportion of the training words.
import math
word_counts = []
for i in (features):
word_counts.append(len(i.split(" ")))
print("Average word counts:", sum(word_counts)/len(word_counts))
# Round max length of padding to the nearest tenth
max_length = math.ceil((sum(word_counts)/len(word_counts))/10) * 10
print("Padded onto length: ", max_length)
# Delete unused variable
del word_counts
Average word counts: 18.223942208462333 Padded onto length: 20
We can then proceed to do the actual padding. This is done for all instances of the training example.
padded = pad_sequences(sequences, maxlen=max_length,
padding='post', truncating='post')
for i in range(5):
print("Sequence of length", len(sequences[i]),
"was padded onto", len(padded[i]))
Sequence of length 9 was padded onto 20 Sequence of length 16 was padded onto 20 Sequence of length 29 was padded onto 20 Sequence of length 36 was padded onto 20 Sequence of length 35 was padded onto 20
Generally, oversampling makes fake data so that the minority labels are not squashed by the majority.
Constraint of using SMOTE within this dataset is that it's very fragile to classes with only one example. Since SMOTE works via k-nearest neighbors, at least two members of a class would be mandatory for it to work. For this project, we're going to use ROS.
ROS generates random data as opposed to the better, more superior SMOTE which generates "more credible" data. We can't use SMOTE on all sheets because of the n_neighbors and distance constraint.
We find the least common multiple of all class members amount, and oversample from there to get the matrix dimensions uniform.
Synthetic samples generated with respect to the most amount of classes in all columns so that input shape (features and labels) are consistent against each other.
import copy
from collections import defaultdict, Counter
from imblearn.over_sampling import SMOTE, RandomOverSampler
max_amount = np.lcm.reduce([1, 2, 3, 4, 5, 6, 7]) * 40
labels_processed = np.zeros((max_amount, 8))
for i in range(8):
unique_classes = np.unique(df[df.columns[1:][i]])
amount = [int(
(max_amount/len(unique_classes)))
] * len(unique_classes)
x = {k:v for (k,v) in zip(unique_classes, amount)}
smt = RandomOverSampler(sampling_strategy=x)
X, y = smt.fit_resample(padded, labels[:,i])
labels_processed[:,i] = y
print("Column", i, "done.")
# We don't save nor use the X variable between each iteration,
# instead only using the last one.
print("Shape of input features is:", X.shape) # 20 words represented on 20 columns
print("Shape of labels is:",labels_processed.shape) # 8 different classes on 8 columns
Column 0 done. Column 1 done. Column 2 done. Column 3 done. Column 4 done. Column 5 done. Column 6 done. Column 7 done. Shape of input features is: (16800, 20) Shape of labels is: (16800, 8)
# Sanity check time!
for i in range(8):
print("For column", df.columns[1:][i])
print(Counter(labels_processed[:, i]), "\n------")
print("First ten entries of the processed labels:\n", labels_processed[:10, :])
print("First ten entries of the processed features:\n", X[:10, :])
# Last and final assertion to catch matrix errors
try:
assert(X.shape[0] == labels_processed.shape[0])
print("Assertion completed!")
except AssertionError:
print("Error!\nShape of X is", X.shape,
"while labels is", labels_processed.shape)
raise AssertionError
For column supportive
Counter({4.0: 2800, 6.0: 2800, 7.0: 2800, 5.0: 2800, 2.0: 2800, 3.0: 2800})
------
For column easy
Counter({4.0: 2800, 5.0: 2800, 3.0: 2800, 6.0: 2800, 7.0: 2800, 2.0: 2800})
------
For column efficient
Counter({4.0: 3360, 5.0: 3360, 6.0: 3360, 3.0: 3360, 7.0: 3360})
------
For column clear
Counter({3.0: 2800, 6.0: 2800, 7.0: 2800, 4.0: 2800, 5.0: 2800, 2.0: 2800})
------
For column exciting
Counter({4.0: 2800, 3.0: 2800, 5.0: 2800, 6.0: 2800, 7.0: 2800, 2.0: 2800})
------
For column interesting
Counter({4.0: 2800, 3.0: 2800, 5.0: 2800, 6.0: 2800, 2.0: 2800, 7.0: 2800})
------
For column inventive
Counter({4.0: 2800, 3.0: 2800, 2.0: 2800, 5.0: 2800, 6.0: 2800, 1.0: 2800})
------
For column leading edge
Counter({4.0: 2800, 3.0: 2800, 5.0: 2800, 2.0: 2800, 6.0: 2800, 1.0: 2800})
------
First ten entries of the processed labels:
[[4. 4. 4. 3. 4. 4. 4. 4.]
[6. 4. 5. 6. 4. 4. 3. 4.]
[7. 5. 6. 7. 4. 4. 4. 4.]
[5. 3. 4. 4. 3. 3. 3. 3.]
[6. 4. 6. 7. 5. 5. 3. 4.]
[6. 5. 6. 7. 4. 4. 3. 3.]
[7. 6. 6. 6. 6. 6. 3. 3.]
[7. 5. 6. 6. 5. 5. 3. 3.]
[6. 4. 6. 5. 3. 5. 4. 5.]
[5. 5. 5. 5. 3. 4. 2. 2.]]
First ten entries of the processed features:
[[ 2 5 3 672 2 32 95 673 49 0 0 0 0 0
0 0 0 0 0 0]
[ 2 45 3 876 2 336 6 2 1240 154 674 22 337 6
1241 223 0 0 0 0]
[ 69 877 2 30 15 7 1242 16 8 878 879 192 96 6
880 15 7 17 4 448]
[ 2 10 3 7 34 143 882 224 178 38 675 38 2 12
122 37 4 2 389 1243]
[ 69 877 2 30 15 7 537 16 8 878 879 192 96 6
880 15 7 17 4 677]
[ 2 10 3 155 23 62 70 41 7 391 22 105 19 2
10 14 7 17 4 1245]
[ 2 10 3 7 261 9 539 680 96 6 56 19 3 7
17 4 300 2 32 6]
[ 2 10 3 1246 450 4 262 101 4 124 30 13 2 32
0 0 0 0 0 0]
[ 2 5 3 29 8 1247 540 23 883 11 2 338 6 2
681 9 2 30 339 0]
[ 2 5 3 32 225 13 8 1248 339 22 25 0 0 0
0 0 0 0 0 0]]
Assertion completed!
Finally we get to this part. We can use sklearn's train_test_split to do the usual splitting for test set (validation set is specified within building the model in the next section).
Also, specifying random_state ensures reproducibility.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, labels_processed, test_size=0.2, random_state=42)
# Sanity check again and again
print("Training examples:", X_train.shape)
print("Training labels:", y_train.shape)
print("Test examples:", X_test.shape)
print("Test labels:", y_test.shape)
Training examples: (13440, 20) Training labels: (13440, 8) Test examples: (3360, 20) Test labels: (3360, 8)
Since we're training on augmented data, we're going to approach this as conventional regression compared to the LSTM approach we're doing it before.
LSTM model in this manner would result in a pretty bad metric score (around 0.5 to 0.7 F1-score) whereas our fully-connected dense layers resulted in a pretty satisfying F1, which is around 0.8 to 1 for each column.
# Wrapper functions for result interpretation
from matplotlib import pyplot as plt
def plot_graphs(history, string):
plt.plot(history.history[string])
plt.plot(history.history['val_'+string])
plt.xlabel("Epochs")
plt.ylabel(string)
plt.legend([string, 'val_'+string])
plt.show()
def transform_results(estimates):
res = []
for x in estimates:
if x > 7:
x = 7
res.append(x)
elif x < 0:
x = 0
res.append(x)
else:
res.append(x)
return res
from tensorflow.keras import layers
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Adam
from sklearn.metrics import classification_report
initializer = tf.keras.initializers.RandomNormal(seed=1) # Reproducibility
model = Sequential(name="classifier_NN")
model.add(tf.keras.Input(shape=(X_train.shape[1],)))
# Automatically detect features as input nodes
model.add(layers.Dense(256, activation='relu', kernel_initializer=initializer))
# model.add(layers.Dropout(0.1))
model.add(layers.Dense(256, activation='relu', kernel_initializer=initializer))
# model.add(layers.Dropout(0.1))
model.add(layers.Dense(256, activation='relu', kernel_initializer=initializer))
model.add(layers.Dropout(0.1))
model.add(layers.Dense(256, activation='relu', kernel_initializer=initializer))
model.add(layers.Dropout(0.1))
model.add(layers.Dense(128, activation='relu', kernel_initializer=initializer))
model.add(layers.Dropout(0.1))
model.add(layers.Dense(128, activation='relu', kernel_initializer=initializer))
model.add(layers.Dropout(0.1))
model.add(layers.Dense(128, activation='relu', kernel_initializer=initializer))
model.add(layers.Dropout(0.15))
model.add(layers.Dense(8)) # output layer
opt = tf.keras.optimizers.Adam(learning_rate=0.0005)
model.compile(
loss=tf.keras.losses.MeanSquaredError(), metrics=['mae'], optimizer=opt)
model.summary()
Model: "classifier_NN"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 256) 5376
dense_1 (Dense) (None, 256) 65792
dense_2 (Dense) (None, 256) 65792
dropout (Dropout) (None, 256) 0
dense_3 (Dense) (None, 256) 65792
dropout_1 (Dropout) (None, 256) 0
dense_4 (Dense) (None, 128) 32896
dropout_2 (Dropout) (None, 128) 0
dense_5 (Dense) (None, 128) 16512
dropout_3 (Dropout) (None, 128) 0
dense_6 (Dense) (None, 128) 16512
dropout_4 (Dropout) (None, 128) 0
dense_7 (Dense) (None, 8) 1032
=================================================================
Total params: 269,704
Trainable params: 269,704
Non-trainable params: 0
_________________________________________________________________
es = tf.keras.callbacks.EarlyStopping(
monitor='val_loss', mode='min', verbose=1, patience=15)
history = model.fit(
X_train, y_train, epochs=150, batch_size=32,
validation_split=0.3,
callbacks=[es],
verbose=1)
plot_graphs(history, "mae")
plot_graphs(history, "loss")
Epoch 1/150 294/294 [==============================] - 4s 6ms/step - loss: 3.6967 - mae: 1.4628 - val_loss: 2.0927 - val_mae: 1.1289 Epoch 2/150 294/294 [==============================] - 2s 5ms/step - loss: 1.4785 - mae: 0.9073 - val_loss: 1.0444 - val_mae: 0.7746 Epoch 3/150 294/294 [==============================] - 2s 5ms/step - loss: 0.7985 - mae: 0.6681 - val_loss: 0.4674 - val_mae: 0.4950 Epoch 4/150 294/294 [==============================] - 2s 5ms/step - loss: 0.6082 - mae: 0.5784 - val_loss: 0.3572 - val_mae: 0.4350 Epoch 5/150 294/294 [==============================] - 2s 5ms/step - loss: 0.5199 - mae: 0.5281 - val_loss: 0.3091 - val_mae: 0.3936 Epoch 6/150 294/294 [==============================] - 2s 5ms/step - loss: 0.4663 - mae: 0.4959 - val_loss: 0.4037 - val_mae: 0.4666 Epoch 7/150 294/294 [==============================] - 2s 5ms/step - loss: 0.4585 - mae: 0.4872 - val_loss: 0.3016 - val_mae: 0.3911 Epoch 8/150 294/294 [==============================] - 2s 5ms/step - loss: 0.4145 - mae: 0.4598 - val_loss: 0.2836 - val_mae: 0.3750 Epoch 9/150 294/294 [==============================] - 2s 5ms/step - loss: 0.3805 - mae: 0.4373 - val_loss: 0.2711 - val_mae: 0.3660 Epoch 10/150 294/294 [==============================] - 2s 5ms/step - loss: 0.3656 - mae: 0.4260 - val_loss: 0.2779 - val_mae: 0.3645 Epoch 11/150 294/294 [==============================] - 2s 5ms/step - loss: 0.3360 - mae: 0.4060 - val_loss: 0.2357 - val_mae: 0.3220 Epoch 12/150 294/294 [==============================] - 2s 5ms/step - loss: 0.3214 - mae: 0.3965 - val_loss: 0.2114 - val_mae: 0.2926 Epoch 13/150 294/294 [==============================] - 2s 5ms/step - loss: 0.3159 - mae: 0.3904 - val_loss: 0.2605 - val_mae: 0.3414 Epoch 14/150 294/294 [==============================] - 2s 5ms/step - loss: 0.3088 - mae: 0.3831 - val_loss: 0.2005 - val_mae: 0.2773 Epoch 15/150 294/294 [==============================] - 2s 5ms/step - loss: 0.3079 - mae: 0.3828 - val_loss: 0.1950 - val_mae: 0.2716 Epoch 16/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2894 - mae: 0.3683 - val_loss: 0.1924 - val_mae: 0.2685 Epoch 17/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2840 - mae: 0.3610 - val_loss: 0.2075 - val_mae: 0.2683 Epoch 18/150 294/294 [==============================] - 1s 5ms/step - loss: 0.2870 - mae: 0.3625 - val_loss: 0.2078 - val_mae: 0.2857 Epoch 19/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2722 - mae: 0.3512 - val_loss: 0.2088 - val_mae: 0.2914 Epoch 20/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2750 - mae: 0.3526 - val_loss: 0.2467 - val_mae: 0.3384 Epoch 21/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2652 - mae: 0.3469 - val_loss: 0.2143 - val_mae: 0.2984 Epoch 22/150 294/294 [==============================] - 1s 5ms/step - loss: 0.2725 - mae: 0.3483 - val_loss: 0.1893 - val_mae: 0.2668 Epoch 23/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2527 - mae: 0.3343 - val_loss: 0.1946 - val_mae: 0.2709 Epoch 24/150 294/294 [==============================] - 1s 5ms/step - loss: 0.2655 - mae: 0.3382 - val_loss: 0.1933 - val_mae: 0.2632 Epoch 25/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2470 - mae: 0.3285 - val_loss: 0.1935 - val_mae: 0.2793 Epoch 26/150 294/294 [==============================] - 1s 5ms/step - loss: 0.2455 - mae: 0.3282 - val_loss: 0.1843 - val_mae: 0.2577 Epoch 27/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2530 - mae: 0.3319 - val_loss: 0.1868 - val_mae: 0.2396 Epoch 28/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2502 - mae: 0.3277 - val_loss: 0.1893 - val_mae: 0.2552 Epoch 29/150 294/294 [==============================] - 1s 5ms/step - loss: 0.2422 - mae: 0.3220 - val_loss: 0.1905 - val_mae: 0.2660 Epoch 30/150 294/294 [==============================] - 1s 5ms/step - loss: 0.2317 - mae: 0.3125 - val_loss: 0.1750 - val_mae: 0.2270 Epoch 31/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2465 - mae: 0.3209 - val_loss: 0.1818 - val_mae: 0.2482 Epoch 32/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2307 - mae: 0.3102 - val_loss: 0.1738 - val_mae: 0.2426 Epoch 33/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2276 - mae: 0.3055 - val_loss: 0.1721 - val_mae: 0.2328 Epoch 34/150 294/294 [==============================] - 1s 5ms/step - loss: 0.2235 - mae: 0.3049 - val_loss: 0.1636 - val_mae: 0.2261 Epoch 35/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2256 - mae: 0.3033 - val_loss: 0.1699 - val_mae: 0.2195 Epoch 36/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2307 - mae: 0.3080 - val_loss: 0.1682 - val_mae: 0.2233 Epoch 37/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2180 - mae: 0.2965 - val_loss: 0.1644 - val_mae: 0.2117 Epoch 38/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2244 - mae: 0.3004 - val_loss: 0.1702 - val_mae: 0.2240 Epoch 39/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2416 - mae: 0.3105 - val_loss: 0.1801 - val_mae: 0.2223 Epoch 40/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2264 - mae: 0.3035 - val_loss: 0.1606 - val_mae: 0.2009 Epoch 41/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2149 - mae: 0.2944 - val_loss: 0.1666 - val_mae: 0.2182 Epoch 42/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2177 - mae: 0.2943 - val_loss: 0.1572 - val_mae: 0.2016 Epoch 43/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2132 - mae: 0.2902 - val_loss: 0.1693 - val_mae: 0.2131 Epoch 44/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2132 - mae: 0.2921 - val_loss: 0.1583 - val_mae: 0.1946 Epoch 45/150 294/294 [==============================] - 1s 5ms/step - loss: 0.2115 - mae: 0.2883 - val_loss: 0.1660 - val_mae: 0.2335 Epoch 46/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2066 - mae: 0.2833 - val_loss: 0.1577 - val_mae: 0.2158 Epoch 47/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2070 - mae: 0.2813 - val_loss: 0.1644 - val_mae: 0.2282 Epoch 48/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2287 - mae: 0.2969 - val_loss: 0.1694 - val_mae: 0.2140 Epoch 49/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2144 - mae: 0.2865 - val_loss: 0.1699 - val_mae: 0.2217 Epoch 50/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2108 - mae: 0.2831 - val_loss: 0.1556 - val_mae: 0.2113 Epoch 51/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2052 - mae: 0.2775 - val_loss: 0.1544 - val_mae: 0.1925 Epoch 52/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2012 - mae: 0.2760 - val_loss: 0.1523 - val_mae: 0.1877 Epoch 53/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2584 - mae: 0.3096 - val_loss: 0.1837 - val_mae: 0.2534 Epoch 54/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2396 - mae: 0.2994 - val_loss: 0.1952 - val_mae: 0.2553 Epoch 55/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2304 - mae: 0.2934 - val_loss: 0.1756 - val_mae: 0.2246 Epoch 56/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2274 - mae: 0.2889 - val_loss: 0.1596 - val_mae: 0.2064 Epoch 57/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2167 - mae: 0.2842 - val_loss: 0.1624 - val_mae: 0.2134 Epoch 58/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2126 - mae: 0.2786 - val_loss: 0.1745 - val_mae: 0.2056 Epoch 59/150 294/294 [==============================] - 2s 5ms/step - loss: 0.1998 - mae: 0.2703 - val_loss: 0.1584 - val_mae: 0.1894 Epoch 60/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2027 - mae: 0.2721 - val_loss: 0.1568 - val_mae: 0.1933 Epoch 61/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2002 - mae: 0.2724 - val_loss: 0.1566 - val_mae: 0.2114 Epoch 62/150 294/294 [==============================] - 2s 5ms/step - loss: 0.1946 - mae: 0.2675 - val_loss: 0.1530 - val_mae: 0.1780 Epoch 63/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2007 - mae: 0.2696 - val_loss: 0.2029 - val_mae: 0.2266 Epoch 64/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2125 - mae: 0.2802 - val_loss: 0.1667 - val_mae: 0.2000 Epoch 65/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2125 - mae: 0.2799 - val_loss: 0.1733 - val_mae: 0.2244 Epoch 66/150 294/294 [==============================] - 2s 5ms/step - loss: 0.2026 - mae: 0.2748 - val_loss: 0.1637 - val_mae: 0.2068 Epoch 67/150 294/294 [==============================] - 2s 5ms/step - loss: 0.1970 - mae: 0.2676 - val_loss: 0.1597 - val_mae: 0.2063 Epoch 00067: early stopping
Here we evaluate on y_test, which we made sure earlier comes from a similar distribution within the training set itself.
Resulted on a pretty confident mae and mse, which I would say a good result considering the number of classes we have.
scores = model.evaluate(
X_test, y_test, batch_size=1, verbose=1, sample_weight=None, steps=None,
callbacks=None, max_queue_size=10, workers=4, use_multiprocessing=False,
return_dict=False
)
print("------------------ EVALUATION FINISHED! ------------------".center(115))
for i in range(len(scores)):
print("%s: %.2f%%" % (model.metrics_names[i], scores[i]*100))
3360/3360 [==============================] - 11s 3ms/step - loss: 0.1553 - mae: 0.2034
------------------ EVALUATION FINISHED! ------------------
loss: 15.53%
mae: 20.34%
Due to the multi-dimensional nature of our results and test set, we need to reshape the predictions by using numpy's flatten() attribute, which flattens the array column-wise if we specify the argument order='F' to flatten in Fortran-style order.
Keep in mind this will indeed aggregate the metrics like on the classification_report() below.
After that, we're rounding the result to the nearest integer -- because we're tackling this originally as a regression task. For example, if a prediction is 4.7, it will be rounded to 5, and therefore will be predicted as class no. 5.
Multi-class ordinal classification cannot be implemented with standard fully-connected layer. And implementing multi-headed layers are outside the scope of this project.
from sklearn.metrics import classification_report
y_pred = model.predict(X_test)
y_pred_flattened = y_pred.flatten(order='F')
y_test_flattened = y_test.flatten(order='F')
y_classified = ([round(float(x)) for x in y_pred_flattened])
print("First ten examples of the test set:")
print("Truth labels:", y_test_flattened[:10])
print("Predicted labels:", y_classified[:10])
print("Overall performance on all columns:\n",
classification_report(y_classified, y_test_flattened))
First ten examples of the test set:
Truth labels: [3. 2. 3. 2. 5. 4. 4. 4. 4. 7.]
Predicted labels: [3, 2, 3, 2, 5, 4, 4, 4, 4, 2]
Overall performance on all columns:
precision recall f1-score support
1 1.00 1.00 1.00 1088
2 0.99 0.99 0.99 3870
3 0.93 0.95 0.94 4601
4 0.81 0.85 0.83 4562
5 0.84 0.77 0.81 5104
6 0.86 0.85 0.86 4415
7 0.93 0.95 0.94 3240
accuracy 0.89 26880
macro avg 0.91 0.91 0.91 26880
weighted avg 0.89 0.89 0.89 26880
# Wrapper function(s) from earlier notebook
from tensorflow.keras.utils import to_categorical
from sklearn.metrics import roc_curve, auc, roc_auc_score
import seaborn as sns
import matplotlib.pyplot as plt
avgroc = dict()
def plot_multiclass_roc_avg(fpr, tpr, n_classes, figsize=(17, 6)):
roc_auc = dict() # Initialize empty dictionary to store scores
target_names = [1, 2, 3, 4, 5, 6, 7] # Target names to predict
for i in range(n_classes): # Loop over the number of classes
roc_auc[i] = auc(fpr[i], tpr[i]) # Calculate AUC with respect to false and true positive rate
# Initialize empty figure to render our plots on
fig, ax = plt.subplots(figsize=figsize)
ax.plot([0, 1], [0, 1], 'k--')
ax.set_xlim([0.0, 1.0])
ax.set_ylim([0.0, 1.05])
ax.set_xlabel('False Positive Rate')
ax.set_ylabel('True Positive Rate')
ax.set_title('Receiver operating characteristic')
# Render each AUC curve, 1 for each class
for i in range(len(target_names)):
ax.plot(fpr[i], tpr[i],
label='ROC curve (area = %0.2f) for %s' % (roc_auc[i],
target_names[i]))
ax.legend(loc="best")
ax.grid(alpha=.4)
sns.despine()
plt.show()
def plot_multiclass_roc(clf, X_test, y_test, n_classes, figsize=(17, 6)):
"""
Modified function behavior, now predicts column-wise
"""
y_score = clf.predict_proba(X_test)
target_names = np.unique(y_test)
# structures
fpr = dict()
tpr = dict()
roc_auc = dict()
# calculate dummies once
y_test_dummies = pd.get_dummies(y_test, drop_first=False).values
# print(y_test_dummies.shape, y_score.shape) # Uncomment to debug and see shapes (we feared an input mismatch)
for i in range(n_classes):
fpr[i], tpr[i], _ = roc_curve(y_test_dummies[:, i], y_score[:, i]) # Get the true and positive rate for each class prediction(s)
roc_auc[i] = auc(fpr[i], tpr[i]) # Calculate AUC with respect to false and true positive rate
# Render each AUC curve, 1 for each class
fig, ax = plt.subplots(figsize=figsize)
ax.plot([0, 1], [0, 1], 'k--')
ax.set_xlim([0.0, 1.0])
ax.set_ylim([0.0, 1.05])
ax.set_xlabel('False Positive Rate')
ax.set_ylabel('True Positive Rate')
ax.set_title('Receiver operating characteristic')
for i in range(len(target_names)):
ax.plot(fpr[i], tpr[i], label='ROC curve (area = %0.2f) for %s'
% (roc_auc[i], target_names[i]))
ax.legend(loc="best")
ax.grid(alpha=.4)
sns.despine()
plt.show()
return fpr, tpr
def plot_multiclass_roc_LSTM(X_test, y_test,
n_classes, figsize=(17, 6), model=model):
"""
Custom-made function to plot ROC curve.
Sorry I can't run because I don't have the
excel file used for this particular notebook.
"""
y_score = to_categorical(y_test)
target_names = ["obstructive vs supportive",
"complicated vs easy", "inefficient vs efficient",
" confusing vs clear", "boring vs exciting",
"not interesting vs interesting",
"conventional vs inventive", "usual vs leading edge"] # Initialize target names
# (from original data)
# structures
fpr = dict()
tpr = dict()
roc_auc = dict()
# One-liner transformation of our existing result.
# This reshapes our input from ordinal to categorical (one-hot encoded)
# and pads the input with zeroes to handle input mismatch.
y_test_dummies = np.pad(
pd.get_dummies(y_classified, drop_first=False).values,
((0, 0), (min(labels.flatten()), 0)))
# print(y_test_dummies.shape, y_score.shape)
# Quick debug purposes after one-hot encoding. It can go wrong in so many ways.
# Mainly done to ensure matrix dimension is right.
for i in range(1, n_classes):
fpr[i], tpr[i], _ = roc_curve(y_test_dummies[:, i], y_score[:, i])
roc_auc[i] = auc(fpr[i], tpr[i])
# Render each AUC curve, 1 for each class
fig, ax = plt.subplots(figsize=figsize)
ax.plot([0, 1], [0, 1], 'k--')
ax.set_xlim([0.0, 1.0])
ax.set_ylim([0.0, 1.05])
ax.set_xlabel('False Positive Rate')
ax.set_ylabel('True Positive Rate')
ax.set_title('Column-wise Receiver Operating Characteristic Plot')
for i in range(1, len(target_names)):
ax.plot(fpr[i], tpr[i], label='ROC curve (area = %0.2f) for %s'
% (roc_auc[i], target_names[i]))
ax.legend(loc="best")
ax.grid(alpha=.4)
sns.despine()
plt.show()
from sklearn.preprocessing import LabelBinarizer
from sklearn.metrics import roc_curve, auc, roc_auc_score
# Function to compute area under curve.
# AUC is computed via binary one-vs-all situation.
def score_auc(y_test, y_pred, average="weighted"):
lb = LabelBinarizer()
lb.fit(y_test)
y_test = lb.transform(y_test)
y_pred = lb.transform(y_pred)
return roc_auc_score(y_test, y_pred, average=average)
fig, c_ax = plt.subplots(1, 1, figsize=(12, 8))
# Function to compute area under curve, but for multi-class (not multi-label)
# AUC is computed via binary one-vs-all situation.
def multiclass_roc_auc_score(y_test, y_pred, average="weighted"):
lb = LabelBinarizer()
lb.fit(y_test)
y_test = lb.transform(y_test)
y_pred = lb.transform(y_pred)
for (idx, c_label) in enumerate(set(labels.flatten())):
fpr, tpr, thresholds = roc_curve(
y_test[:, idx].astype(int), y_pred[:, idx])
c_ax.plot(fpr, tpr, label='%s (AUC:%0.2f)' % (c_label, auc(fpr, tpr)))
c_ax.plot(fpr, fpr, 'b-', label='Random Guessing')
c_ax.plot(fpr, tpr, 'r-', label='Average')
return roc_auc_score(y_test, y_pred, average=average)
print('ROC AUC score:',
multiclass_roc_auc_score(y_test_flattened, y_classified))
c_ax.legend()
c_ax.set_xlabel('False Positive Rate')
c_ax.set_ylabel('True Positive Rate')
c_ax.set_title('Aggregated Class-wise Receiver Operating Characteristic Plot')
plt.savefig("auc.png")
plt.show()
ROC AUC score: 0.9358068899750152
After plotting the ROC curve, we can initialize some empty list to keep track of the evaluation metrics we're going to use.
from xgboost import XGBClassifier
from sklearn.svm import SVC, LinearSVC
from operator import add
acc = []
pre = []
recall = []
f1 = []
rms = []
aucscore = []
mae = []
from sklearn.metrics import confusion_matrix, f1_score, precision_score
from sklearn.metrics import recall_score, classification_report
from sklearn.metrics import plot_confusion_matrix, accuracy_score
from sklearn.metrics import mean_squared_error, recall_score
from sklearn.metrics import mean_absolute_error
# making an empty DataFrame to log results
data = {'metric':['Accuracy', 'Precision', 'Recall', 'F1 Score','RMSE','AUC score', 'MAE']}
metric_df = pd.DataFrame(data).set_index(['metric'])
# metric_df # Uncomment metric_df to take a preview of the dataframe.
We can also print classification report from each column, and append each resulting evaluation metric to the existing list we initialized.
y_pred = model.predict(X_test)
for i in range(8): # Loop over columns
print("Column", df.columns[1:][i]) # Print which column we're analyzing
# Rounding the predictions because we're approaching a problem as a regression task
# despite we have 7 output nodes. This way, regression outputs, which would be continuous,
# are seen as classes instead.
estimates = [round(float(x)) for x in y_pred[:, i]]
estimates = transform_results(estimates)
# print(np.unique(estimates)) # Uncomment to see the unique classes in that column
print(classification_report(y_test[:, i], estimates), "\n------") # Print classification report
# Append all the scores to our previously initialized list
acc.append(accuracy_score(y_test[:, i], estimates))
pre.append(precision_score(y_test[:, i], estimates, average='weighted'))
recall.append(recall_score(y_test[:, i], estimates, average='weighted'))
f1.append(f1_score(y_test[:, i], estimates, average='weighted'))
rms.append(np.sqrt(mean_squared_error(y_test[:, i], estimates)))
aucscore.append(score_auc(y_test[:, i], estimates))
mae.append(mean_absolute_error(y_test[:, i], estimates))
Column supportive
precision recall f1-score support
2.0 1.00 1.00 1.00 544
3.0 0.97 0.96 0.97 576
4.0 0.85 0.80 0.82 594
5.0 0.69 0.72 0.71 571
6.0 0.73 0.74 0.74 541
7.0 0.89 0.90 0.90 534
accuracy 0.85 3360
macro avg 0.86 0.85 0.86 3360
weighted avg 0.86 0.85 0.85 3360
------
Column easy
precision recall f1-score support
2.0 1.00 1.00 1.00 544
3.0 0.97 0.96 0.97 575
4.0 0.85 0.82 0.84 589
5.0 0.74 0.76 0.75 573
6.0 0.80 0.82 0.81 541
7.0 0.96 0.97 0.96 538
accuracy 0.89 3360
macro avg 0.89 0.89 0.89 3360
weighted avg 0.89 0.89 0.89 3360
------
Column efficient
precision recall f1-score support
3.0 0.98 0.81 0.89 670
4.0 0.74 0.65 0.69 681
5.0 0.61 0.92 0.74 726
6.0 0.79 0.72 0.75 639
7.0 0.98 0.82 0.90 644
accuracy 0.79 3360
macro avg 0.82 0.79 0.79 3360
weighted avg 0.82 0.79 0.79 3360
------
Column clear
precision recall f1-score support
2.0 1.00 1.00 1.00 544
3.0 0.97 0.96 0.96 578
4.0 0.85 0.80 0.82 595
5.0 0.69 0.72 0.71 567
6.0 0.74 0.75 0.74 540
7.0 0.90 0.91 0.91 536
accuracy 0.86 3360
macro avg 0.86 0.86 0.86 3360
weighted avg 0.86 0.86 0.86 3360
------
Column exciting
precision recall f1-score support
2.0 1.00 1.00 1.00 545
3.0 0.95 0.97 0.96 574
4.0 0.84 0.83 0.83 572
5.0 0.85 0.80 0.82 602
6.0 0.91 0.94 0.92 533
7.0 0.98 0.99 0.99 534
accuracy 0.92 3360
macro avg 0.92 0.92 0.92 3360
weighted avg 0.92 0.92 0.92 3360
------
Column interesting
precision recall f1-score support
2.0 1.00 1.00 1.00 545
3.0 0.95 0.97 0.96 574
4.0 0.85 0.84 0.85 578
5.0 0.85 0.82 0.84 592
6.0 0.91 0.93 0.92 537
7.0 0.98 0.99 0.99 534
accuracy 0.92 3360
macro avg 0.92 0.93 0.93 3360
weighted avg 0.92 0.92 0.92 3360
------
Column inventive
precision recall f1-score support
1.0 1.00 1.00 1.00 545
2.0 0.98 0.98 0.98 576
3.0 0.91 0.89 0.90 589
4.0 0.90 0.88 0.89 586
5.0 0.94 0.97 0.96 533
6.0 0.98 1.00 0.99 531
accuracy 0.95 3360
macro avg 0.95 0.95 0.95 3360
weighted avg 0.95 0.95 0.95 3360
------
Column leading edge
precision recall f1-score support
1.0 1.00 1.00 1.00 544
2.0 0.98 0.98 0.98 576
3.0 0.93 0.93 0.93 582
4.0 0.95 0.93 0.94 589
5.0 0.97 0.99 0.98 536
6.0 0.99 1.00 0.99 533
accuracy 0.97 3360
macro avg 0.97 0.97 0.97 3360
weighted avg 0.97 0.97 0.97 3360
------
Here we display the average of each evaluation metric by calling np.mean() on each array -- remember, each array is comprised of multiple evaluation metrics; one for each column.
# Printing every classification metric
print("Average Accuracy : {}".format(np.mean(acc)))
print("Average Precision : {}".format(np.mean(pre)))
print("Average Recall : {}".format(np.mean(recall)))
print("Average F1-score : {}".format(np.mean(f1)))
print("Average RMSE-score : {}".format(np.mean(rms)))
print("Average MAE : {}".format(np.mean(mae)))
print("Average AUC-score : {}".format(np.mean(aucscore)))
metric_df['Neural Network Classifier'] = [
np.mean(acc), np.mean(pre), np.mean(recall), np.mean(f1),
np.mean(rms), np.mean(aucscore), np.mean(mae)]
Average Accuracy : 0.8934523809523809 Average Precision : 0.8975077117271504 Average Recall : 0.8934523809523809 Average F1-score : 0.8940450224613606 Average RMSE-score : 0.38376328122233827 Average MAE : 0.12269345238095238 Average AUC-score : 0.9351214174575726
We can export the model to HDF5 format designed to store large amounts of information. More info:
model.save('UX_Regression.h5')
Our metric_df now looks like this for one single method. Note that we're trying to look for the best methods; so let's continue working.
metric_df
| Neural Network Classifier | |
|---|---|
| metric | |
| Accuracy | 0.893452 |
| Precision | 0.897508 |
| Recall | 0.893452 |
| F1 Score | 0.894045 |
| RMSE | 0.383763 |
| AUC score | 0.935121 |
| MAE | 0.122693 |
For comparison, we try implementing XGB.
ffpo = dict(), dict()
temp = []
from scipy import interp # Interpolation package
target_label_names = ["obstructive vs supportive",
"complicated vs easy",
"inefficient vs efficient",
" confusing vs clear",
"boring vs exciting",
"not interesting vs interesting",
"conventional vs inventive",
"usual vs leading edge"]
def plot_multiclass_roc_optim(clf, X_train,
y_train, X_test, y_test, figsize=(17, 6)):
"""
Modified function behavior, now predicts column-wise
"""
# y_score = clf.predict_proba(X_test)
target_names = np.unique(y_test)
# structures
fpr = dict()
fprt = dict()
tprt = dict()
tpr = dict()
roc_auc = dict()
mean_tpr = dict()
all_fpr = dict()
# X_train, X_test, y_train[:,i], y_test[:,i],
# calculate dummies once
for j in range(8):
# Again, this is to convert ordinal to categorical (one-hot encoded)
y_test_dummies = pd.get_dummies(y_test[:, j], drop_first=False).values
n_classes = y_test_dummies.shape[1]-1 # NumPy sees 9 classes -- because Python is zero-indexed
# Loop over those 8 classes
for i in range(n_classes):
clf.fit(X_train, y_train[:, j]) # Fit the XGB model for that 1 class
y_score = clf.predict_proba(X_test)
fpr[i], tpr[i], _ = roc_curve(
y_test_dummies[:, i], y_score[:, i]) # calculate ROC for that 1 class
all_fpr[j] = np.unique(
np.concatenate([fpr[i] for i in range(n_classes)]))
mean_tpr[j] = np.zeros_like(all_fpr[j])
for i in range(n_classes):
fpr[i], tpr[i], _ = roc_curve(y_test_dummies[:, i], y_score[:, i])
mean_tpr[j] += interp(all_fpr[j], fpr[i], tpr[i])
mean_tpr[j] /= n_classes
fprt[j] = all_fpr[j]
tprt[j] = mean_tpr[j]
roc_auc[j] = auc(fprt[j], tprt[j])
# roc for each class
fig, ax = plt.subplots(figsize=figsize)
ax.plot([0, 1], [0, 1], 'k--')
ax.set_xlim([0.0, 1.0])
ax.set_ylim([0.0, 1.05])
ax.set_xlabel('False Positive Rate')
ax.set_ylabel('True Positive Rate')
ax.set_title('Receiver operating characteristic')
for j in range(8):
ax.plot(fprt[j], tprt[j],
label="ROC_curve Average of label {0} (area = {1:0.2f})"
.format(target_label_names[j], roc_auc[j]))
ax.legend(loc="best")
ax.grid(alpha=.4)
sns.despine()
plt.show()
To provide brevity, we can define a wrapper so that we need to only invoke this function when we want to plot the resulting predictions across all different lea2rning algorithms.
We need only to call this plot_result_model() function and pass in the specified keyword arguments to:
.fit().So, we can expect that this wrapper function will be used to infer the results across different learning algorithms.
Keep in mind that the function plot_result_model() will be again wrapped within plot_all_result_model()
ffpo = dict(), dict()
temp = []
def plot_result_model(clf, X_train, X_test, y_train, y_test,
n_classes, ffpo, ttpo):
clf.fit(X_train, y_train) # Fit our classifier `clf` of choice.
# predictions over test set
predictions = [round(float(x)) for x in clf.predict(X_test)] # Again, rounding the predictions
# because we're approaching a problem as a regression task
print((classification_report(y_test, predictions))) # Print classification report
# calculating Accuracy Score, Precision Score, Recall Score and F1 Score.
# In the end, we'll append them to our existing list to record.
print('Confusion Matrix: \n', confusion_matrix(y_test, predictions))
acc.append(accuracy_score(y_test, predictions))
print(f'Accuracy Score : {accuracy_score(y_test, predictions)}')
pre.append(precision_score(y_test, predictions, average='weighted'))
print('Precision Score : ' + str(
precision_score(y_test, predictions, average='weighted')))
recal.append(recall_score(y_test,predictions, average='weighted'))
print('Recall Score : ' + str(recall_score(
y_test, predictions, average='weighted')))
f1.append(f1_score(y_test, predictions, average='weighted'))
print('F1 Score : ' + str(f1_score(
y_test, predictions, average='weighted')))
rms.append(np.sqrt(mean_squared_error(y_test, predictions)))
aucscore.append(score_auc(y_test, predictions))
mae.append(mean_absolute_error(y_test, predictions))
fpo,tpo = plot_multiclass_roc(
clf, X_test, y_test, n_classes)
ffpo.append(fpo)
ttpo.append(tpo)
# Wrapper for the above wrapper function to plot all columns instead of one column
def plot_all_result_model(clf, X_test, y_train, y_test,
ffpo, ttpo):
for i in range(8):
print('For column:', df.columns[1:][i])
n_classes = len(np.unique(df[df.columns[1:][i]]))
plot_result_model(clf,X_train, X_test, y_train[:, i],
y_test[:, i], n_classes, ffpo, ttpo)
# Again, initialize empty lists
acc = []
pre = []
recal = []
f1 = []
rms = []
aucscore = []
mae = []
fpo = dict()
tpo = dict()
ffpo = []
ttpo = []
clf = XGBClassifier(eval_metric='mlogloss')
plot_all_result_model(clf, X_test, y_train, y_test,
ffpo, ttpo)
For column: supportive
precision recall f1-score support
2.0 0.99 1.00 1.00 544
3.0 0.80 0.99 0.88 576
4.0 0.74 0.65 0.69 594
5.0 0.62 0.60 0.61 571
6.0 0.68 0.60 0.64 541
7.0 0.88 0.90 0.89 534
accuracy 0.79 3360
macro avg 0.78 0.79 0.78 3360
weighted avg 0.78 0.79 0.78 3360
Confusion Matrix:
[[543 1 0 0 0 0]
[ 0 568 7 0 1 0]
[ 3 74 389 97 25 6]
[ 0 40 70 341 110 10]
[ 0 23 55 88 322 53]
[ 1 8 8 20 14 483]]
Accuracy Score : 0.7875
Precision Score : 0.7821350428031527
Recall Score : 0.7875
F1 Score : 0.7820890074156168
For column: easy
precision recall f1-score support
2.0 0.99 1.00 1.00 544
3.0 0.79 0.99 0.87 575
4.0 0.70 0.62 0.66 589
5.0 0.65 0.57 0.61 573
6.0 0.73 0.69 0.71 541
7.0 0.94 0.97 0.95 538
accuracy 0.80 3360
macro avg 0.80 0.81 0.80 3360
weighted avg 0.80 0.80 0.80 3360
Confusion Matrix:
[[543 1 0 0 0 0]
[ 0 567 6 2 0 0]
[ 3 77 365 111 27 6]
[ 0 47 77 328 110 11]
[ 0 27 70 55 373 16]
[ 1 3 3 7 4 520]]
Accuracy Score : 0.8023809523809524
Precision Score : 0.7965338961527718
Recall Score : 0.8023809523809524
F1 Score : 0.796529517166695
For column: efficient
precision recall f1-score support
3.0 0.97 0.81 0.89 670
4.0 0.63 0.76 0.69 681
5.0 0.61 0.64 0.63 726
6.0 0.66 0.68 0.67 639
7.0 0.95 0.82 0.88 644
accuracy 0.74 3360
macro avg 0.76 0.74 0.75 3360
weighted avg 0.76 0.74 0.75 3360
Confusion Matrix:
[[545 125 0 0 0]
[ 10 520 123 25 3]
[ 4 112 464 131 15]
[ 1 51 145 433 9]
[ 1 20 23 69 531]]
Accuracy Score : 0.7419642857142857
Precision Score : 0.7613349936383595
Recall Score : 0.7419642857142857
F1 Score : 0.7479655591667455
For column: clear
precision recall f1-score support
2.0 1.00 1.00 1.00 544
3.0 0.78 0.98 0.87 578
4.0 0.70 0.63 0.66 595
5.0 0.64 0.61 0.62 567
6.0 0.71 0.59 0.64 540
7.0 0.88 0.91 0.89 536
accuracy 0.79 3360
macro avg 0.78 0.79 0.78 3360
weighted avg 0.78 0.79 0.78 3360
Confusion Matrix:
[[543 1 0 0 0 0]
[ 0 568 8 1 1 0]
[ 0 80 377 107 25 6]
[ 0 41 70 347 98 11]
[ 1 25 73 72 319 50]
[ 0 9 13 19 7 488]]
Accuracy Score : 0.7863095238095238
Precision Score : 0.7814073587228159
Recall Score : 0.7863095238095238
F1 Score : 0.7807370651805632
For column: exciting
precision recall f1-score support
2.0 0.99 1.00 0.99 545
3.0 0.79 0.97 0.87 574
4.0 0.68 0.61 0.65 572
5.0 0.70 0.59 0.64 602
6.0 0.80 0.80 0.80 533
7.0 0.96 0.99 0.97 534
accuracy 0.82 3360
macro avg 0.82 0.83 0.82 3360
weighted avg 0.82 0.82 0.82 3360
Confusion Matrix:
[[543 1 1 0 0 0]
[ 1 559 5 7 2 0]
[ 0 82 351 105 31 3]
[ 4 52 107 356 69 14]
[ 0 11 49 41 425 7]
[ 0 0 1 0 4 529]]
Accuracy Score : 0.8223214285714285
Precision Score : 0.81673630572397
Recall Score : 0.8223214285714285
F1 Score : 0.8167717495152512
For column: interesting
precision recall f1-score support
2.0 1.00 1.00 1.00 545
3.0 0.79 0.98 0.87 574
4.0 0.71 0.63 0.67 578
5.0 0.73 0.62 0.67 592
6.0 0.81 0.81 0.81 537
7.0 0.95 0.99 0.97 534
accuracy 0.83 3360
macro avg 0.83 0.84 0.83 3360
weighted avg 0.83 0.83 0.83 3360
Confusion Matrix:
[[543 1 1 0 0 0]
[ 1 560 5 6 2 0]
[ 0 77 363 95 40 3]
[ 0 58 92 369 55 18]
[ 0 14 46 37 433 7]
[ 0 1 1 0 3 529]]
Accuracy Score : 0.8324404761904762
Precision Score : 0.8283878324184464
Recall Score : 0.8324404761904762
F1 Score : 0.8274399558120317
For column: inventive
precision recall f1-score support
1.0 1.00 1.00 1.00 545
2.0 0.81 1.00 0.90 576
3.0 0.78 0.66 0.71 589
4.0 0.74 0.67 0.70 586
5.0 0.83 0.82 0.83 533
6.0 0.96 1.00 0.98 531
accuracy 0.85 3360
macro avg 0.85 0.86 0.85 3360
weighted avg 0.85 0.85 0.85 3360
Confusion Matrix:
[[544 1 0 0 0 0]
[ 0 574 2 0 0 0]
[ 0 73 387 90 34 5]
[ 0 47 80 393 54 12]
[ 0 11 28 49 438 7]
[ 0 0 0 0 0 531]]
Accuracy Score : 0.8532738095238095
Precision Score : 0.8502087763661068
Recall Score : 0.8532738095238095
F1 Score : 0.8488751879259745
For column: leading edge
precision recall f1-score support
1.0 1.00 1.00 1.00 544
2.0 0.83 1.00 0.91 576
3.0 0.76 0.72 0.74 582
4.0 0.78 0.66 0.71 589
5.0 0.84 0.82 0.83 536
6.0 0.96 1.00 0.98 533
accuracy 0.86 3360
macro avg 0.86 0.87 0.86 3360
weighted avg 0.86 0.86 0.86 3360
Confusion Matrix:
[[544 0 0 0 0 0]
[ 0 576 0 0 0 0]
[ 0 55 420 72 30 5]
[ 0 49 90 387 52 11]
[ 0 11 40 40 440 5]
[ 0 0 0 0 0 533]]
Accuracy Score : 0.8630952380952381
Precision Score : 0.860110621156973
Recall Score : 0.8630952380952381
F1 Score : 0.8592648329169004
Here we plot the average ROC for each class, found with XGBoost.
xgb_classifier = XGBClassifier(eval_metric='mlogloss')
plot_multiclass_roc_optim(xgb_classifier,X_train, y_train, X_test, y_test)
Here we display the average of each evaluation metric by calling np.mean() on each array -- remember, each array is comprised of multiple evaluation metrics; one for each column.
print("Average Accuracy : {}".format(np.mean(acc)))
print("Average Precision : {}".format(np.mean(pre)))
print("Average Recall : {}".format(np.mean(recal)))
print("Average F1-score : {}".format(np.mean(f1)))
print("Average RMSE-score : {}".format(np.mean(rms)))
print("Average AUC-score : {}".format(np.mean(aucscore)))
metric_df['XGB'] = [np.mean(acc), np.mean(pre), np.mean(recal),
np.mean(f1), np.mean(rms), np.mean(aucscore), np.mean(mae)]
Average Accuracy : 0.8111607142857142 Average Precision : 0.8096068533728245 Average Recall : 0.8111607142857142 Average F1-score : 0.8074591093874722 Average RMSE-score : 0.6290541897573358 Average AUC-score : 0.8854706028238077
Another comparison would be the SVM (Support Vector Machine) algorithm. We try and see whether this shallow learning algorithm can still predict correctly across all 8 classes compared to 2 previous implementations.
acc = []
pre = []
recal = []
mae = []
f1 = []
rms = []
aucscore = []
fpo = dict()
tpo = dict()
ffpo = []
ttpo = []
clf = SVC(probability=True) # Make SVC classifier
plot_all_result_model(clf, X_test, y_train, y_test,
ffpo, ttpo) # Invoke function
For column: supportive
precision recall f1-score support
2.0 0.91 1.00 0.95 544
3.0 0.69 0.92 0.79 576
4.0 0.69 0.61 0.65 594
5.0 0.66 0.54 0.60 571
6.0 0.71 0.57 0.63 541
7.0 0.85 0.90 0.87 534
accuracy 0.76 3360
macro avg 0.75 0.76 0.75 3360
weighted avg 0.75 0.76 0.75 3360
Confusion Matrix:
[[543 1 0 0 0 0]
[ 16 531 8 0 0 21]
[ 19 115 363 71 18 8]
[ 3 76 86 310 89 7]
[ 17 37 58 70 307 52]
[ 2 6 8 19 16 483]]
Accuracy Score : 0.7550595238095238
Precision Score : 0.7495408068826231
Recall Score : 0.7550595238095238
F1 Score : 0.7462240773730214
For column: easy
precision recall f1-score support
2.0 0.90 1.00 0.95 544
3.0 0.69 0.92 0.79 575
4.0 0.70 0.59 0.64 589
5.0 0.70 0.55 0.62 573
6.0 0.77 0.67 0.72 541
7.0 0.91 0.97 0.94 538
accuracy 0.78 3360
macro avg 0.78 0.78 0.78 3360
weighted avg 0.78 0.78 0.77 3360
Confusion Matrix:
[[543 1 0 0 0 0]
[ 16 530 5 2 1 21]
[ 21 116 350 78 18 6]
[ 4 75 87 315 85 7]
[ 15 45 58 45 363 15]
[ 3 1 2 8 4 520]]
Accuracy Score : 0.7800595238095238
Precision Score : 0.7766847967563638
Recall Score : 0.7800595238095238
F1 Score : 0.7721354601545184
For column: efficient
precision recall f1-score support
3.0 0.89 0.82 0.85 670
4.0 0.56 0.76 0.64 681
5.0 0.60 0.58 0.59 726
6.0 0.72 0.62 0.66 639
7.0 0.94 0.82 0.88 644
accuracy 0.72 3360
macro avg 0.74 0.72 0.73 3360
weighted avg 0.74 0.72 0.72 3360
Confusion Matrix:
[[550 116 0 0 4]
[ 34 515 113 12 7]
[ 12 204 424 71 15]
[ 17 71 152 393 6]
[ 8 19 16 70 531]]
Accuracy Score : 0.7181547619047619
Precision Score : 0.7370578979044515
Recall Score : 0.7181547619047619
F1 Score : 0.7227175884834174
For column: clear
precision recall f1-score support
2.0 0.90 1.00 0.95 544
3.0 0.68 0.92 0.78 578
4.0 0.72 0.61 0.66 595
5.0 0.67 0.56 0.61 567
6.0 0.72 0.59 0.65 540
7.0 0.86 0.91 0.88 536
accuracy 0.76 3360
macro avg 0.76 0.76 0.75 3360
weighted avg 0.76 0.76 0.75 3360
Confusion Matrix:
[[543 1 0 0 0 0]
[ 16 531 7 2 1 21]
[ 21 120 360 71 17 6]
[ 3 79 70 316 92 7]
[ 17 37 51 70 317 48]
[ 2 11 10 15 11 487]]
Accuracy Score : 0.7601190476190476
Precision Score : 0.7566584009870992
Recall Score : 0.7601190476190476
F1 Score : 0.7519532504174382
For column: exciting
precision recall f1-score support
2.0 0.91 1.00 0.95 545
3.0 0.69 0.93 0.79 574
4.0 0.68 0.57 0.62 572
5.0 0.78 0.59 0.67 602
6.0 0.87 0.79 0.83 533
7.0 0.94 0.99 0.97 534
accuracy 0.81 3360
macro avg 0.81 0.81 0.80 3360
weighted avg 0.81 0.81 0.80 3360
Confusion Matrix:
[[543 2 0 0 0 0]
[ 16 536 3 3 2 14]
[ 23 129 327 69 20 4]
[ 2 87 112 356 37 8]
[ 10 28 38 30 420 7]
[ 0 0 1 0 4 529]]
Accuracy Score : 0.8068452380952381
Precision Score : 0.8079037739951445
Recall Score : 0.8068452380952381
F1 Score : 0.8003565363814342
For column: interesting
precision recall f1-score support
2.0 0.90 1.00 0.95 545
3.0 0.69 0.93 0.79 574
4.0 0.72 0.61 0.66 578
5.0 0.80 0.61 0.69 592
6.0 0.87 0.77 0.82 537
7.0 0.94 0.99 0.97 534
accuracy 0.81 3360
macro avg 0.82 0.82 0.81 3360
weighted avg 0.82 0.81 0.81 3360
Confusion Matrix:
[[543 2 0 0 0 0]
[ 17 533 6 3 1 14]
[ 21 123 353 55 21 5]
[ 3 91 94 364 33 7]
[ 17 27 39 35 412 7]
[ 0 0 1 0 4 529]]
Accuracy Score : 0.8136904761904762
Precision Score : 0.8167933078022993
Recall Score : 0.8136904761904762
F1 Score : 0.80834388537611
For column: inventive
precision recall f1-score support
1.0 0.91 1.00 0.95 545
2.0 0.71 0.94 0.80 576
3.0 0.75 0.63 0.68 589
4.0 0.83 0.66 0.73 586
5.0 0.89 0.80 0.85 533
6.0 0.94 1.00 0.97 531
accuracy 0.83 3360
macro avg 0.84 0.84 0.83 3360
weighted avg 0.84 0.83 0.83 3360
Confusion Matrix:
[[544 1 0 0 0 0]
[ 16 539 7 0 0 14]
[ 22 120 369 58 16 4]
[ 2 78 80 385 35 6]
[ 17 26 36 19 428 7]
[ 0 0 0 0 0 531]]
Accuracy Score : 0.8321428571428572
Precision Score : 0.83563138687498
Recall Score : 0.8321428571428572
F1 Score : 0.8274518485967612
For column: leading edge
precision recall f1-score support
1.0 0.92 1.00 0.96 544
2.0 0.70 0.94 0.80 576
3.0 0.77 0.64 0.70 582
4.0 0.85 0.69 0.76 589
5.0 0.92 0.81 0.86 536
6.0 0.95 1.00 0.97 533
accuracy 0.84 3360
macro avg 0.85 0.85 0.84 3360
weighted avg 0.85 0.84 0.84 3360
Confusion Matrix:
[[544 0 0 0 0 0]
[ 16 540 6 0 0 14]
[ 22 116 372 55 13 4]
[ 2 81 68 408 24 6]
[ 10 32 37 16 436 5]
[ 0 0 0 0 0 533]]
Accuracy Score : 0.8431547619047619
Precision Score : 0.8488672811410958
Recall Score : 0.8431547619047619
F1 Score : 0.8396789268921511
Here we plot the average ROC for each class, found with SVM.
clf = SVC(probability=True)
plot_multiclass_roc_optim(clf, X_train, y_train, X_test, y_test)
Here we display the average of each evaluation metric by calling np.mean() on each array -- remember, each array is comprised of multiple evaluation metrics; one for each column.
print("Average Accuracy : {}".format(np.mean(acc)))
print("Average Precision : {}".format(np.mean(pre)))
print("Average Recall : {}".format(np.mean(recal)))
print("Average F1-score : {}".format(np.mean(f1)))
print("Average RMSE score : {}".format(np.mean(rms)))
print("Average AUC score : {}".format(np.mean(aucscore)))
metric_df['SVM'] = [np.mean(acc), np.mean(pre), np.mean(recal),
np.mean(f1), np.mean(rms), np.mean(aucscore), np.mean(mae)]
Average Accuracy : 0.7886532738095238 Average Precision : 0.7911422065430072 Average Recall : 0.7886532738095238 Average F1-score : 0.7836076967093565 Average RMSE score : 0.7839059574870244 Average AUC score : 0.8718729141730868
Credit: Muhammad for Towards Data Science
Ordinal classifier basically does a similar thing to one-hot encoding -- but despite having one value each column, ordinal encoding maps unique binary values for each of our unique classes.
We implement a custom object called OrdinalClassifier built on top of any sklearn model that supports the predict_proba() function; ranging from Naive Bayes to decision trees.
Last implementation, we deduced the labels by one, from [1,2,3,4,5,6,7] to [0,1,2,3,4,5,6] due to the nature of zero-indexing of standard sklearn library. But this doesn't work if the labels aren't ranging fully from 1 to 7 -- could be 2 to 6 or 2 to 7 instead.
So the solution is to change the overall behavior of the OrdinalClassifier object to take maximum class (e.g. 7) and minimum class (e.g. 2 or 3) into consideration. From there, we simulate the np.argmax() function to be added the minimum class before being returned. We also add the minimum class as an index towards self.clfs dictionary when adding new value, so we don't need to deduce the labels by hand. In other words, labels stay as-is instead of being deduced by 1 :)
This also changes how the approach for ordinal encoding works -- instead of k - 1 like the author's original implementation, we feed k numbers of encoding because our labels are already one-indexed.
Keep in mind that the predict_proba() function does not yet implement this change, because it's only a wrapper function before eventually calling predict() in which we care the most about the result.
class OrdinalClassifier():
def __init__(self, clf):
self.clf = clf
self.clfs = {}
def fit(self, X, y):
self.unique_class = np.sort(np.unique(y))
self.max_class = max(self.unique_class)
self.min_class = min(self.unique_class)
if self.unique_class.shape[0] > 2:
# print(self.unique_class.shape[0]) # Uncomment to debug
for i in range(self.unique_class.shape[0]):
# for each k ordinal value
# we fit a binary classification problem
binary_y = (y > self.unique_class[i]).astype(np.uint8)
clf = clone(self.clf)
clf.fit(X, binary_y)
self.clfs[i+self.min_class] = clf
def predict_proba(self, X):
clfs_predict = {
k:self.clfs[k].predict_proba(X) for k in self.clfs
}
# print(self.clfs) # Uncomment to debug
predicted = []
for i,y in enumerate(self.unique_class):
if i == 0:
# V1 = 1 - Pr(y > V1)
predicted.append(1-clfs_predict[y][:, 1])
elif y in clfs_predict:
# Vi = Pr(y > Vi-1) - Pr(y > Vi)
predicted.append(
clfs_predict[y-1][:, 1] - clfs_predict[y][:, 1])
else:
# Vk = Pr(y > Vk-1)
predicted.append(clfs_predict[y-1][:, 1])
return np.vstack(predicted).T
def predict(self, X):
return np.argmax(
self.predict_proba(X), axis=1) + self.min_class
from sklearn.base import clone
acc = []
pre = []
recal = []
f1 = []
mae = []
rms = []
aucscore = []
fpo = dict()
tpo = dict()
ffpo = []
ttpo = []
clf = OrdinalClassifier(XGBClassifier(eval_metric='mlogloss')) # Make a classifier
plot_all_result_model(clf, X_test, y_train, y_test, # Invoke the function from before
ffpo, ttpo)
For column: supportive
precision recall f1-score support
2.0 1.00 1.00 1.00 544
3.0 0.84 0.98 0.90 576
4.0 0.63 0.61 0.62 594
5.0 0.54 0.49 0.51 571
6.0 0.65 0.61 0.63 541
7.0 0.89 0.90 0.90 534
accuracy 0.76 3360
macro avg 0.76 0.77 0.76 3360
weighted avg 0.75 0.76 0.76 3360
Confusion Matrix:
[[543 1 0 0 0 0]
[ 0 567 6 2 1 0]
[ 0 51 362 147 31 3]
[ 0 31 123 278 134 5]
[ 0 23 75 63 331 49]
[ 1 6 8 23 13 483]]
Accuracy Score : 0.763095238095238
Precision Score : 0.7549960331953902
Recall Score : 0.763095238095238
F1 Score : 0.7576200197273073
For column: easy
precision recall f1-score support
2.0 1.00 1.00 1.00 544
3.0 0.82 0.98 0.89 575
4.0 0.64 0.60 0.62 589
5.0 0.59 0.49 0.54 573
6.0 0.65 0.66 0.65 541
7.0 0.96 0.97 0.96 538
accuracy 0.78 3360
macro avg 0.78 0.78 0.78 3360
weighted avg 0.77 0.78 0.77 3360
Confusion Matrix:
[[543 1 0 0 0 0]
[ 0 566 5 2 2 0]
[ 0 58 355 121 54 1]
[ 0 30 122 281 134 6]
[ 0 33 69 67 357 15]
[ 1 3 2 6 6 520]]
Accuracy Score : 0.7803571428571429
Precision Score : 0.7723402898083773
Recall Score : 0.7803571428571429
F1 Score : 0.7741605688739556
For column: efficient
precision recall f1-score support
3.0 0.97 0.82 0.89 670
4.0 0.62 0.75 0.68 681
5.0 0.57 0.57 0.57 726
6.0 0.61 0.67 0.64 639
7.0 0.96 0.82 0.89 644
accuracy 0.72 3360
macro avg 0.75 0.73 0.73 3360
weighted avg 0.74 0.72 0.73 3360
Confusion Matrix:
[[548 122 0 0 0]
[ 12 508 129 30 2]
[ 2 126 415 173 10]
[ 0 49 152 430 8]
[ 1 17 27 68 531]]
Accuracy Score : 0.7238095238095238
Precision Score : 0.7447397148696598
Recall Score : 0.7238095238095238
F1 Score : 0.7304128708284927
For column: clear
precision recall f1-score support
2.0 1.00 1.00 1.00 544
3.0 0.83 0.98 0.90 578
4.0 0.65 0.62 0.64 595
5.0 0.55 0.52 0.54 567
6.0 0.62 0.58 0.60 540
7.0 0.90 0.91 0.90 536
accuracy 0.77 3360
macro avg 0.76 0.77 0.76 3360
weighted avg 0.76 0.77 0.76 3360
Confusion Matrix:
[[543 1 0 0 0 0]
[ 0 567 8 1 2 0]
[ 0 47 369 133 45 1]
[ 0 29 112 294 126 6]
[ 1 28 64 88 311 48]
[ 0 8 11 16 14 487]]
Accuracy Score : 0.7651785714285714
Precision Score : 0.7578606523535675
Recall Score : 0.7651785714285714
F1 Score : 0.7601586427123241
For column: exciting
precision recall f1-score support
2.0 1.00 1.00 1.00 545
3.0 0.81 0.97 0.88 574
4.0 0.64 0.51 0.57 572
5.0 0.62 0.57 0.60 602
6.0 0.70 0.77 0.73 533
7.0 0.98 0.99 0.98 534
accuracy 0.79 3360
macro avg 0.79 0.80 0.79 3360
weighted avg 0.79 0.79 0.79 3360
Confusion Matrix:
[[543 2 0 0 0 0]
[ 1 559 8 4 2 0]
[ 0 73 289 146 63 1]
[ 0 37 110 343 107 5]
[ 0 21 41 56 408 7]
[ 0 0 1 0 4 529]]
Accuracy Score : 0.7949404761904761
Precision Score : 0.7873575282996464
Recall Score : 0.7949404761904761
F1 Score : 0.7879298634645988
For column: interesting
precision recall f1-score support
2.0 1.00 1.00 1.00 545
3.0 0.82 0.98 0.89 574
4.0 0.65 0.53 0.59 578
5.0 0.64 0.58 0.61 592
6.0 0.73 0.80 0.76 537
7.0 0.99 0.99 0.99 534
accuracy 0.81 3360
macro avg 0.80 0.81 0.81 3360
weighted avg 0.80 0.81 0.80 3360
Confusion Matrix:
[[543 2 0 0 0 0]
[ 1 560 6 4 3 0]
[ 0 69 306 146 56 1]
[ 0 31 115 342 99 5]
[ 0 21 40 42 432 2]
[ 0 0 1 0 4 529]]
Accuracy Score : 0.8071428571428572
Precision Score : 0.8000854252348313
Recall Score : 0.8071428571428572
F1 Score : 0.8007383798462784
For column: inventive
precision recall f1-score support
1.0 1.00 1.00 1.00 545
2.0 0.84 1.00 0.91 576
3.0 0.74 0.65 0.69 589
4.0 0.66 0.56 0.61 586
5.0 0.73 0.80 0.76 533
6.0 0.98 1.00 0.99 531
accuracy 0.83 3360
macro avg 0.83 0.83 0.83 3360
weighted avg 0.82 0.83 0.82 3360
Confusion Matrix:
[[544 1 0 0 0 0]
[ 0 575 0 0 1 0]
[ 0 54 382 110 43 0]
[ 0 31 110 330 111 4]
[ 0 21 21 58 426 7]
[ 0 0 0 0 0 531]]
Accuracy Score : 0.8297619047619048
Precision Score : 0.8239777080702692
Recall Score : 0.8297619047619048
F1 Score : 0.8242217333465715
For column: leading edge
precision recall f1-score support
1.0 1.00 1.00 1.00 544
2.0 0.85 1.00 0.92 576
3.0 0.76 0.63 0.69 582
4.0 0.69 0.63 0.66 589
5.0 0.75 0.80 0.77 536
6.0 0.98 1.00 0.99 533
accuracy 0.84 3360
macro avg 0.84 0.84 0.84 3360
weighted avg 0.84 0.84 0.84 3360
Confusion Matrix:
[[544 0 0 0 0 0]
[ 0 576 0 0 0 0]
[ 0 49 367 113 53 0]
[ 0 32 86 374 93 4]
[ 0 21 27 55 428 5]
[ 0 0 0 0 0 533]]
Accuracy Score : 0.8398809523809524
Precision Score : 0.8358867020183487
Recall Score : 0.8398809523809524
F1 Score : 0.8353643849188925
Here we plot the average ROC for each class, found with the custom-built Ordinal Classifier we made earlier.
plot_multiclass_roc_optim(clf,X_train, y_train, X_test, y_test)
print("Average Accuracy : {}".format(np.mean(acc)))
print("Average Precision : {}".format(np.mean(pre)))
print("Average Recall : {}".format(np.mean(recal)))
print("Average F1 -score : {}".format(np.mean(f1)))
print("Average RMSE score : {}".format(np.mean(rms)))
print("Average AUC score: {}".format(np.mean(aucscore)))
metric_df['OrdinalModel'] = [np.mean(acc), np.mean(pre),
np.mean(recal), np.mean(f1), np.mean(rms),
np.mean(aucscore), np.mean(mae)]
Average Accuracy : 0.7880208333333334 Average Precision : 0.7846555067312613 Average Recall : 0.7880208333333334 Average F1 -score : 0.7838258079648026 Average RMSE score : 0.6426740709953511 Average AUC score: 0.8715101293824516
Some of the learning algorithms we try and implement other than SVM, XGB, and fully-connected neural networks are as follows:
Bidirectional LSTMs are an extension of traditional LSTMs that can improve model performance on sequence classification problems. In problems where all timesteps of the input sequence are available, Bidirectional LSTMs train two instead of one LSTMs on the input sequence.
Bidirectional LSTMs basically look both ways when training on sequence-based data.
# Callbacks
es = tf.keras.callbacks.EarlyStopping(
monitor='val_loss', mode='min', verbose=1, patience=7)
checkpoint = tf.keras.callbacks.ModelCheckpoint(
"model_checkpoints", monitor="val_loss",
verbose=1, save_best_only=True, mode="min")
# Optimizer
opt = tf.keras.optimizers.Adam(learning_rate=0.001)
LSTM_model = tf.keras.Sequential([
tf.keras.layers.Embedding(2000, 64),
tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)),
# tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(32)),
# use ReLU in place of tanh function
# since they are very good alternatives of each other.
tf.keras.layers.Dense(64, activation='relu',
# Prevent overfitting
kernel_regularizer=tf.keras.regularizers.L2(l2=0.01),
# Ensuring consistent outputs and not run
# into potential local optima
# Avoid initializing standard normal
# (mean = 0, std = 1)
# to reach faster convergence and less initial loss.
kernel_initializer=tf.keras.initializers.RandomNormal(
seed=42)), tf.keras.layers.Dropout(0.3),
tf.keras.layers.Dense(8, activation='linear')])
LSTM_model.summary()
LSTM_model.compile(
loss=tf.keras.losses.MeanSquaredError(),
optimizer=opt, metrics=['mae'])
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
embedding (Embedding) (None, None, 64) 128000
bidirectional (Bidirectiona (None, 128) 66048
l)
dense_8 (Dense) (None, 64) 8256
dropout_5 (Dropout) (None, 64) 0
dense_9 (Dense) (None, 8) 520
=================================================================
Total params: 202,824
Trainable params: 202,824
Non-trainable params: 0
_________________________________________________________________
history = LSTM_model.fit(X_train, y_train,
epochs=100, batch_size=512,
validation_split=0.2,
callbacks=[es],
verbose=1)
Epoch 1/100 21/21 [==============================] - 10s 169ms/step - loss: 18.7021 - mae: 3.8773 - val_loss: 7.0319 - val_mae: 2.1649 Epoch 2/100 21/21 [==============================] - 1s 26ms/step - loss: 5.4854 - mae: 1.8790 - val_loss: 3.1925 - val_mae: 1.4935 Epoch 3/100 21/21 [==============================] - 2s 121ms/step - loss: 4.2390 - mae: 1.6828 - val_loss: 2.8309 - val_mae: 1.4317 Epoch 4/100 21/21 [==============================] - 2s 96ms/step - loss: 3.8136 - mae: 1.6013 - val_loss: 2.5529 - val_mae: 1.3635 Epoch 5/100 21/21 [==============================] - 0s 24ms/step - loss: 3.3333 - mae: 1.4979 - val_loss: 2.0124 - val_mae: 1.2116 Epoch 6/100 21/21 [==============================] - 2s 96ms/step - loss: 2.5625 - mae: 1.2693 - val_loss: 1.3518 - val_mae: 0.9313 Epoch 7/100 21/21 [==============================] - 4s 190ms/step - loss: 1.8875 - mae: 1.0432 - val_loss: 0.8278 - val_mae: 0.6781 Epoch 8/100 21/21 [==============================] - 1s 26ms/step - loss: 1.4312 - mae: 0.8804 - val_loss: 0.6179 - val_mae: 0.5262 Epoch 9/100 21/21 [==============================] - 2s 95ms/step - loss: 1.2355 - mae: 0.8057 - val_loss: 0.5090 - val_mae: 0.4633 Epoch 10/100 21/21 [==============================] - 2s 92ms/step - loss: 1.0696 - mae: 0.7390 - val_loss: 0.4227 - val_mae: 0.3963 Epoch 11/100 21/21 [==============================] - 1s 24ms/step - loss: 0.9629 - mae: 0.6952 - val_loss: 0.3823 - val_mae: 0.3746 Epoch 12/100 21/21 [==============================] - 5s 246ms/step - loss: 0.8875 - mae: 0.6612 - val_loss: 0.3652 - val_mae: 0.3541 Epoch 13/100 21/21 [==============================] - 2s 77ms/step - loss: 0.8289 - mae: 0.6351 - val_loss: 0.3367 - val_mae: 0.3303 Epoch 14/100 21/21 [==============================] - 1s 26ms/step - loss: 0.7886 - mae: 0.6168 - val_loss: 0.3237 - val_mae: 0.3229 Epoch 15/100 21/21 [==============================] - 4s 207ms/step - loss: 0.7556 - mae: 0.6032 - val_loss: 0.2990 - val_mae: 0.2775 Epoch 16/100 21/21 [==============================] - 2s 117ms/step - loss: 0.7211 - mae: 0.5875 - val_loss: 0.2895 - val_mae: 0.2794 Epoch 17/100 21/21 [==============================] - 1s 25ms/step - loss: 0.6960 - mae: 0.5758 - val_loss: 0.2812 - val_mae: 0.2828 Epoch 18/100 21/21 [==============================] - 6s 277ms/step - loss: 0.6783 - mae: 0.5691 - val_loss: 0.2941 - val_mae: 0.2912 Epoch 19/100 21/21 [==============================] - 1s 25ms/step - loss: 0.6558 - mae: 0.5589 - val_loss: 0.2857 - val_mae: 0.3052 Epoch 20/100 21/21 [==============================] - 2s 100ms/step - loss: 0.6346 - mae: 0.5505 - val_loss: 0.2501 - val_mae: 0.2490 Epoch 21/100 21/21 [==============================] - 1s 24ms/step - loss: 0.6274 - mae: 0.5465 - val_loss: 0.3049 - val_mae: 0.3281 Epoch 22/100 21/21 [==============================] - 1s 26ms/step - loss: 0.6246 - mae: 0.5479 - val_loss: 0.2670 - val_mae: 0.2821 Epoch 23/100 21/21 [==============================] - 1s 26ms/step - loss: 0.6096 - mae: 0.5401 - val_loss: 0.2393 - val_mae: 0.2536 Epoch 24/100 21/21 [==============================] - 1s 24ms/step - loss: 0.5989 - mae: 0.5358 - val_loss: 0.2609 - val_mae: 0.2731 Epoch 25/100 21/21 [==============================] - 0s 24ms/step - loss: 0.6081 - mae: 0.5407 - val_loss: 0.2623 - val_mae: 0.3012 Epoch 26/100 21/21 [==============================] - 1s 25ms/step - loss: 0.5934 - mae: 0.5363 - val_loss: 0.3006 - val_mae: 0.3353 Epoch 27/100 21/21 [==============================] - 1s 25ms/step - loss: 0.5803 - mae: 0.5292 - val_loss: 0.2322 - val_mae: 0.2606 Epoch 28/100 21/21 [==============================] - 1s 24ms/step - loss: 0.5609 - mae: 0.5209 - val_loss: 0.2484 - val_mae: 0.2833 Epoch 29/100 21/21 [==============================] - 1s 25ms/step - loss: 0.5527 - mae: 0.5145 - val_loss: 0.2255 - val_mae: 0.2462 Epoch 30/100 21/21 [==============================] - 1s 25ms/step - loss: 0.5481 - mae: 0.5146 - val_loss: 0.2288 - val_mae: 0.2606 Epoch 31/100 21/21 [==============================] - 1s 24ms/step - loss: 0.5358 - mae: 0.5079 - val_loss: 0.2225 - val_mae: 0.2539 Epoch 32/100 21/21 [==============================] - 1s 25ms/step - loss: 0.5308 - mae: 0.5075 - val_loss: 0.2383 - val_mae: 0.2819 Epoch 33/100 21/21 [==============================] - 1s 26ms/step - loss: 0.5388 - mae: 0.5102 - val_loss: 0.2176 - val_mae: 0.2480 Epoch 34/100 21/21 [==============================] - 1s 26ms/step - loss: 0.5261 - mae: 0.5055 - val_loss: 0.2131 - val_mae: 0.2394 Epoch 35/100 21/21 [==============================] - 1s 25ms/step - loss: 0.5290 - mae: 0.5083 - val_loss: 0.2271 - val_mae: 0.2671 Epoch 36/100 21/21 [==============================] - 1s 25ms/step - loss: 0.5157 - mae: 0.5008 - val_loss: 0.2068 - val_mae: 0.2380 Epoch 37/100 21/21 [==============================] - 1s 25ms/step - loss: 0.5124 - mae: 0.4990 - val_loss: 0.2016 - val_mae: 0.2352 Epoch 38/100 21/21 [==============================] - 1s 26ms/step - loss: 0.5092 - mae: 0.5004 - val_loss: 0.2042 - val_mae: 0.2429 Epoch 39/100 21/21 [==============================] - 1s 24ms/step - loss: 0.4922 - mae: 0.4905 - val_loss: 0.2002 - val_mae: 0.2215 Epoch 40/100 21/21 [==============================] - 1s 24ms/step - loss: 0.4939 - mae: 0.4893 - val_loss: 0.2025 - val_mae: 0.2482 Epoch 41/100 21/21 [==============================] - 1s 26ms/step - loss: 0.4954 - mae: 0.4927 - val_loss: 0.2015 - val_mae: 0.2504 Epoch 42/100 21/21 [==============================] - 1s 25ms/step - loss: 0.4949 - mae: 0.4943 - val_loss: 0.2089 - val_mae: 0.2526 Epoch 43/100 21/21 [==============================] - 1s 25ms/step - loss: 0.4710 - mae: 0.4827 - val_loss: 0.1930 - val_mae: 0.2253 Epoch 44/100 21/21 [==============================] - 1s 25ms/step - loss: 0.4758 - mae: 0.4817 - val_loss: 0.1954 - val_mae: 0.2342 Epoch 45/100 21/21 [==============================] - 1s 25ms/step - loss: 0.4698 - mae: 0.4794 - val_loss: 0.1954 - val_mae: 0.2263 Epoch 46/100 21/21 [==============================] - 1s 24ms/step - loss: 0.4749 - mae: 0.4833 - val_loss: 0.1951 - val_mae: 0.2369 Epoch 47/100 21/21 [==============================] - 1s 24ms/step - loss: 0.4781 - mae: 0.4853 - val_loss: 0.1899 - val_mae: 0.2394 Epoch 48/100 21/21 [==============================] - 1s 24ms/step - loss: 0.4758 - mae: 0.4827 - val_loss: 0.2020 - val_mae: 0.2491 Epoch 49/100 21/21 [==============================] - 1s 24ms/step - loss: 0.4643 - mae: 0.4792 - val_loss: 0.1869 - val_mae: 0.2377 Epoch 50/100 21/21 [==============================] - 1s 25ms/step - loss: 0.4681 - mae: 0.4815 - val_loss: 0.1854 - val_mae: 0.2267 Epoch 51/100 21/21 [==============================] - 1s 25ms/step - loss: 0.4593 - mae: 0.4756 - val_loss: 0.1850 - val_mae: 0.2377 Epoch 52/100 21/21 [==============================] - 1s 25ms/step - loss: 0.4662 - mae: 0.4825 - val_loss: 0.1946 - val_mae: 0.2439 Epoch 53/100 21/21 [==============================] - 1s 24ms/step - loss: 0.4641 - mae: 0.4807 - val_loss: 0.1826 - val_mae: 0.2209 Epoch 54/100 21/21 [==============================] - 1s 25ms/step - loss: 0.4502 - mae: 0.4722 - val_loss: 0.1957 - val_mae: 0.2471 Epoch 55/100 21/21 [==============================] - 1s 26ms/step - loss: 0.4480 - mae: 0.4704 - val_loss: 0.1789 - val_mae: 0.2269 Epoch 56/100 21/21 [==============================] - 1s 24ms/step - loss: 0.4453 - mae: 0.4693 - val_loss: 0.1905 - val_mae: 0.2370 Epoch 57/100 21/21 [==============================] - 1s 26ms/step - loss: 0.4452 - mae: 0.4686 - val_loss: 0.1837 - val_mae: 0.2281 Epoch 58/100 21/21 [==============================] - 1s 24ms/step - loss: 0.4425 - mae: 0.4663 - val_loss: 0.1932 - val_mae: 0.2504 Epoch 59/100 21/21 [==============================] - 1s 25ms/step - loss: 0.4402 - mae: 0.4671 - val_loss: 0.1755 - val_mae: 0.2174 Epoch 60/100 21/21 [==============================] - 1s 25ms/step - loss: 0.4410 - mae: 0.4652 - val_loss: 0.1750 - val_mae: 0.2160 Epoch 61/100 21/21 [==============================] - 1s 25ms/step - loss: 0.4461 - mae: 0.4720 - val_loss: 0.2091 - val_mae: 0.2767 Epoch 62/100 21/21 [==============================] - 1s 24ms/step - loss: 0.4411 - mae: 0.4661 - val_loss: 0.1751 - val_mae: 0.2045 Epoch 63/100 21/21 [==============================] - 1s 25ms/step - loss: 0.4314 - mae: 0.4658 - val_loss: 0.1862 - val_mae: 0.2440 Epoch 64/100 21/21 [==============================] - 0s 24ms/step - loss: 0.4347 - mae: 0.4664 - val_loss: 0.1836 - val_mae: 0.2288 Epoch 65/100 21/21 [==============================] - 1s 24ms/step - loss: 0.4291 - mae: 0.4640 - val_loss: 0.1809 - val_mae: 0.2159 Epoch 66/100 21/21 [==============================] - 1s 24ms/step - loss: 0.4301 - mae: 0.4622 - val_loss: 0.1725 - val_mae: 0.2115 Epoch 67/100 21/21 [==============================] - 0s 24ms/step - loss: 0.4192 - mae: 0.4554 - val_loss: 0.1775 - val_mae: 0.2063 Epoch 68/100 21/21 [==============================] - 1s 24ms/step - loss: 0.4168 - mae: 0.4560 - val_loss: 0.1750 - val_mae: 0.2187 Epoch 69/100 21/21 [==============================] - 1s 24ms/step - loss: 0.4194 - mae: 0.4558 - val_loss: 0.1705 - val_mae: 0.2114 Epoch 70/100 21/21 [==============================] - 1s 24ms/step - loss: 0.4216 - mae: 0.4557 - val_loss: 0.1783 - val_mae: 0.2365 Epoch 71/100 21/21 [==============================] - 0s 24ms/step - loss: 0.4168 - mae: 0.4546 - val_loss: 0.1788 - val_mae: 0.2611 Epoch 72/100 21/21 [==============================] - 1s 25ms/step - loss: 0.4236 - mae: 0.4608 - val_loss: 0.1848 - val_mae: 0.2407 Epoch 73/100 21/21 [==============================] - 1s 24ms/step - loss: 0.4130 - mae: 0.4521 - val_loss: 0.1687 - val_mae: 0.2097 Epoch 74/100 21/21 [==============================] - 1s 25ms/step - loss: 0.4150 - mae: 0.4534 - val_loss: 0.1694 - val_mae: 0.2108 Epoch 75/100 21/21 [==============================] - 1s 25ms/step - loss: 0.4160 - mae: 0.4565 - val_loss: 0.1697 - val_mae: 0.2346 Epoch 76/100 21/21 [==============================] - 1s 25ms/step - loss: 0.4216 - mae: 0.4599 - val_loss: 0.1709 - val_mae: 0.2053 Epoch 77/100 21/21 [==============================] - 1s 24ms/step - loss: 0.4131 - mae: 0.4521 - val_loss: 0.1813 - val_mae: 0.2393 Epoch 78/100 21/21 [==============================] - 1s 24ms/step - loss: 0.4148 - mae: 0.4553 - val_loss: 0.1702 - val_mae: 0.2047 Epoch 79/100 21/21 [==============================] - 1s 24ms/step - loss: 0.4176 - mae: 0.4604 - val_loss: 0.1998 - val_mae: 0.2673 Epoch 80/100 21/21 [==============================] - 1s 24ms/step - loss: 0.4148 - mae: 0.4560 - val_loss: 0.1816 - val_mae: 0.2460 Epoch 00080: early stopping
plot_graphs(history, "mae")
plot_graphs(history, "loss")
scores = LSTM_model.evaluate(
X_test, y_test, batch_size=1,
verbose=1, sample_weight=None, steps=None,
callbacks=None, max_queue_size=10, workers=4,
use_multiprocessing=False,
return_dict=False)
print("------------------ EVALUATION FINISHED! ------------------".center(115))
for i in range(len(scores)):
print("%s: %.2f%%" % (model.metrics_names[i], scores[i]*100))
3360/3360 [==============================] - 21s 6ms/step - loss: 0.1745 - mae: 0.2403
------------------ EVALUATION FINISHED! ------------------
loss: 17.45%
mae: 24.03%
Like before, we can initialize some empty list to keep track of the evaluation metrics we're going to use.
acc = []
pre = []
recal = []
mae = []
f1 = []
rms = []
aucscore = []
y_pred = LSTM_model.predict(X_test)
for i in range(8):
print("Column", df.columns[1:][i]) # Print which column we're analyzing
# Rounding the predictions because we're approaching a problem as a regression task
# despite we have 7 output nodes. This way, regression outputs, which would be continuous,
# are seen as classes instead.
estimates = [round(float(x)) for x in y_pred[:, i]]
estimates = transform_results(estimates)
# print(np.unique(estimates)) # Uncomment to see the unique classes in that column
# Append all the scores to our previously initialized list
acc.append(accuracy_score(y_test[:, i], estimates))
pre.append(precision_score(y_test[:, i], estimates, average='weighted'))
recal.append(recall_score(y_test[:, i], estimates, average='weighted'))
f1.append(f1_score(y_test[:, i], estimates, average='weighted'))
rms.append(np.sqrt(mean_squared_error(y_test[:, i], estimates)))
mae.append(mean_absolute_error(y_test[:, i], estimates))
aucscore.append(score_auc(y_test[:, i], estimates))
Column supportive Column easy Column efficient Column clear Column exciting Column interesting Column inventive Column leading edge
print("Average Accuracy : {}".format(np.mean(acc)))
print("Average Precision : {}".format(np.mean(pre)))
print("Average Recall : {}".format(np.mean(recal)))
print("Average F1 score : {}".format(np.mean(f1)))
print("Average RMSE score : {}".format(np.mean(rms)))
print("Average AUC score : {}".format(np.mean(aucscore)))
metric_df['Bidirectional LSTM'] = [np.mean(acc),
np.mean(pre), np.mean(recal),
np.mean(f1),
np.mean(rms), np.mean(aucscore),
np.mean(mae)]
Average Accuracy : 0.9014136904761905 Average Precision : 0.905716502084486 Average Recall : 0.9014136904761905 Average F1 score : 0.902197069922839 Average RMSE score : 0.36563939608155144 Average AUC score : 0.9398928353924897
LSTM_y_pred = LSTM_model.predict(X_test)
LSTM_y_pred_flattened = LSTM_y_pred.flatten(order='F')
LSTM_y_classified = [round(float(x)) for x in LSTM_y_pred_flattened] # Again, rounding the outputs.
print("First ten examples of the test set:")
print("Truth labels:", y_test_flattened[:10])
print("Predicted labels:", LSTM_y_classified[:10])
print("Overall performance on all columns:\n",
classification_report(y_test_flattened, LSTM_y_classified))
First ten examples of the test set:
Truth labels: [3. 2. 3. 2. 5. 4. 4. 4. 4. 7.]
Predicted labels: [3, 2, 3, 2, 5, 4, 4, 4, 4, 2]
Overall performance on all columns:
precision recall f1-score support
1.0 1.00 1.00 1.00 1089
2.0 1.00 1.00 1.00 3874
3.0 0.97 0.95 0.96 4718
4.0 0.85 0.87 0.86 4784
5.0 0.77 0.85 0.81 4700
6.0 0.88 0.82 0.85 4395
7.0 0.96 0.93 0.94 3320
accuracy 0.90 26880
macro avg 0.92 0.91 0.92 26880
weighted avg 0.90 0.90 0.90 26880
In the field of mathematical modeling, a radial basis function network (abbreviated RBFN) is an artificial neural network that uses radial basis functions as activation functions. The output of the network is a linear combination of radial basis functions of the inputs and neuron parameters.
Here, we try to construct a custom RBF class which can then be called like so:
model = Sequential()
model.add(RBFLayer(10,
initializer=InitCentersRandom(X),
betas=1.0,
input_shape=(1,)))
model.add(Dense(1))
from keras import backend as K
from tensorflow.keras.layers import Layer, InputSpec
from keras.initializers import RandomUniform, Initializer, Constant
from sklearn.cluster import KMeans
class InitCentersRandom(Initializer):
""" Initializer for initialization of centers of RBF network
as random samples from the given data set.
Arguments
X: matrix, dataset to choose the centers from (random rows
are taken as centers)
"""
def __init__(self, X):
self.X = X
def __call__(self, shape, dtype=None):
assert shape[1] == self.X.shape[1]
print("done")
idx = np.random.randint(self.X.shape[0], size=shape[0])
return self.X[idx, :]
class RBFLayer(Layer):
""" Layer of Gaussian RBF units.
Arguments
output_dim: number of hidden units (i.e. number of outputs of the
layer)
initializer: instance of initiliazer to initialize centers
betas: float, initial value for betas
"""
def __init__(self, output_dim, initializer=None, betas=1.0, **kwargs):
self.output_dim = output_dim
self.init_betas = betas
if not initializer:
self.initializer = RandomUniform(0.0, 1.0)
else:
self.initializer = initializer
super(RBFLayer, self).__init__(**kwargs)
def build(self, input_shape):
self.centers = self.add_weight(name='centers',
shape=(self.output_dim, input_shape[1]),
initializer=self.initializer,
trainable=True)
self.betas = self.add_weight(name='betas',
shape=(self.output_dim,),
initializer=Constant(
value=self.init_betas),
# initializer='ones',
trainable=True)
super(RBFLayer, self).build(input_shape)
def call(self, x):
C = K.expand_dims(self.centers)
H = K.transpose(C-K.transpose(x))
return K.exp(-self.betas * K.sum(H**2, axis=1))
# C = self.centers[np.newaxis, :, :]
# X = x[:, np.newaxis, :]
# diffnorm = K.sum((C-X)**2, axis=-1)
# ret = K.exp( - self.betas * diffnorm)
# return ret
def compute_output_shape(self, input_shape):
return (input_shape[0], self.output_dim)
def get_config(self):
# have to define get_config to be able to use model_from_json
config = {
'output_dim': self.output_dim
}
base_config = super(RBFLayer, self).get_config()
return dict(list(base_config.items()) + list(config.items()))
class InitCentersKMeans(Initializer):
""" Initializer for initialization of centers of RBF network
by clustering the given data set.
Arguments
X: matrix, dataset
"""
def __init__(self, X, max_iter=100):
self.X = X
self.max_iter = max_iter
def __call__(self, shape, dtype=None):
assert shape[1] == self.X.shape[1]
n_centers = shape[0]
km = KMeans(n_clusters=n_centers, max_iter=self.max_iter, verbose=0)
km.fit(self.X)
return km.cluster_centers_
After defining our RBF class, we can start to build our model based on the RBFLayer class.
from tensorflow.keras.layers import Dense, Activation
model = Sequential()
rbflayer = RBFLayer(100,
initializer=InitCentersKMeans(X_train),
betas=3.0,
input_shape=(X_train.shape[1],)) # Here is our previously initialize RBF class
model.add(rbflayer)
model.add(Dense(8))
model.add(Activation('linear'))
model.compile(loss='mean_squared_error',
optimizer='adam', metrics=['accuracy'])
model.build(X_train.shape)
print(model.summary())
history1 = model.fit(X_train, y_train, epochs=100)
Model: "sequential_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
rbf_layer (RBFLayer) (None, 100) 2100
dense_10 (Dense) (None, 8) 808
activation (Activation) (None, 8) 0
=================================================================
Total params: 2,908
Trainable params: 2,908
Non-trainable params: 0
_________________________________________________________________
None
Epoch 1/100
420/420 [==============================] - 2s 3ms/step - loss: 19.9404 - accuracy: 0.4516
Epoch 2/100
420/420 [==============================] - 1s 3ms/step - loss: 16.7170 - accuracy: 0.4763
Epoch 3/100
420/420 [==============================] - 1s 3ms/step - loss: 13.9596 - accuracy: 0.4764
Epoch 4/100
420/420 [==============================] - 1s 3ms/step - loss: 11.6118 - accuracy: 0.4769
Epoch 5/100
420/420 [==============================] - 1s 3ms/step - loss: 9.6296 - accuracy: 0.4797
Epoch 6/100
420/420 [==============================] - 1s 3ms/step - loss: 7.9742 - accuracy: 0.4798
Epoch 7/100
420/420 [==============================] - 1s 3ms/step - loss: 6.6119 - accuracy: 0.4798
Epoch 8/100
420/420 [==============================] - 1s 3ms/step - loss: 5.5105 - accuracy: 0.4798
Epoch 9/100
420/420 [==============================] - 1s 3ms/step - loss: 4.6402 - accuracy: 0.4803
Epoch 10/100
420/420 [==============================] - 1s 3ms/step - loss: 3.9730 - accuracy: 0.4810
Epoch 11/100
420/420 [==============================] - 1s 3ms/step - loss: 3.4801 - accuracy: 0.4815
Epoch 12/100
420/420 [==============================] - 1s 3ms/step - loss: 3.1323 - accuracy: 0.4818
Epoch 13/100
420/420 [==============================] - 1s 3ms/step - loss: 2.9001 - accuracy: 0.4828
Epoch 14/100
420/420 [==============================] - 1s 3ms/step - loss: 2.7552 - accuracy: 0.4831
Epoch 15/100
420/420 [==============================] - 1s 3ms/step - loss: 2.6726 - accuracy: 0.4831
Epoch 16/100
420/420 [==============================] - 1s 3ms/step - loss: 2.6301 - accuracy: 0.4824
Epoch 17/100
420/420 [==============================] - 1s 3ms/step - loss: 2.6106 - accuracy: 0.4825
Epoch 18/100
420/420 [==============================] - 1s 3ms/step - loss: 2.6027 - accuracy: 0.4825
Epoch 19/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5996 - accuracy: 0.4825
Epoch 20/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5982 - accuracy: 0.4825
Epoch 21/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5974 - accuracy: 0.4825
Epoch 22/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5967 - accuracy: 0.4825
Epoch 23/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5963 - accuracy: 0.4825
Epoch 24/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5960 - accuracy: 0.4825
Epoch 25/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5958 - accuracy: 0.4825
Epoch 26/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5956 - accuracy: 0.4828
Epoch 27/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5955 - accuracy: 0.4823
Epoch 28/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5953 - accuracy: 0.4824
Epoch 29/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5954 - accuracy: 0.4824
Epoch 30/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5953 - accuracy: 0.4822
Epoch 31/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5953 - accuracy: 0.4826
Epoch 32/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4825
Epoch 33/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5953 - accuracy: 0.4828
Epoch 34/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4822
Epoch 35/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5953 - accuracy: 0.4825
Epoch 36/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4827
Epoch 37/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4824
Epoch 38/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4824
Epoch 39/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4830
Epoch 40/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5953 - accuracy: 0.4824
Epoch 41/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5953 - accuracy: 0.4824
Epoch 42/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4825
Epoch 43/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5951 - accuracy: 0.4826
Epoch 44/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5953 - accuracy: 0.4823
Epoch 45/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4824
Epoch 46/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4827
Epoch 47/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4828
Epoch 48/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5951 - accuracy: 0.4826
Epoch 49/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5955 - accuracy: 0.4825
Epoch 50/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4824
Epoch 51/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4828
Epoch 52/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4821
Epoch 53/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5953 - accuracy: 0.4821
Epoch 54/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5954 - accuracy: 0.4824
Epoch 55/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4824
Epoch 56/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4828
Epoch 57/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5953 - accuracy: 0.4827
Epoch 58/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4822
Epoch 59/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5953 - accuracy: 0.4825
Epoch 60/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4824
Epoch 61/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4824
Epoch 62/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5953 - accuracy: 0.4827
Epoch 63/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4825
Epoch 64/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5951 - accuracy: 0.4827
Epoch 65/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4827
Epoch 66/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4822
Epoch 67/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5951 - accuracy: 0.4826
Epoch 68/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4823
Epoch 69/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5951 - accuracy: 0.4825
Epoch 70/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4822
Epoch 71/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5953 - accuracy: 0.4823
Epoch 72/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4827
Epoch 73/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4825
Epoch 74/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4823
Epoch 75/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4827
Epoch 76/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4824
Epoch 77/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4827
Epoch 78/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5950 - accuracy: 0.4825
Epoch 79/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5951 - accuracy: 0.4828
Epoch 80/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4825
Epoch 81/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5953 - accuracy: 0.4825
Epoch 82/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5953 - accuracy: 0.4826
Epoch 83/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4823
Epoch 84/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4826
Epoch 85/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5951 - accuracy: 0.4822
Epoch 86/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4825
Epoch 87/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5953 - accuracy: 0.4825
Epoch 88/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5953 - accuracy: 0.4825
Epoch 89/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4827
Epoch 90/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5953 - accuracy: 0.4822
Epoch 91/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5953 - accuracy: 0.4826
Epoch 92/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4824
Epoch 93/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4828
Epoch 94/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4822
Epoch 95/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4822
Epoch 96/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4827
Epoch 97/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5953 - accuracy: 0.4826
Epoch 98/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5953 - accuracy: 0.4827
Epoch 99/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4824
Epoch 100/100
420/420 [==============================] - 1s 3ms/step - loss: 2.5952 - accuracy: 0.4823
scores = model.evaluate(
X_test, y_test, batch_size=1,
verbose=1, sample_weight=None, steps=None,
callbacks=None, max_queue_size=10,
workers=4, use_multiprocessing=False,
return_dict=False)
print("------------------ EVALUATION FINISHED! ------------------".center(115))
for i in range(len(scores)):
print("%s: %.2f%%" % (model.metrics_names[i], scores[i]*100))
3360/3360 [==============================] - 9s 3ms/step - loss: 2.5107 - accuracy: 0.4818
------------------ EVALUATION FINISHED! ------------------
loss: 251.07%
accuracy: 48.18%
Like before, we can initialize some empty list to keep track of the evaluation metrics we're going to use.
acc = []
pre = []
recal = []
mae = []
f1 = []
rms = []
aucscore = []
y_pred = model.predict(X_test)
for i in range(8): # Loop over columns
print("Column", df.columns[1:][i]) # Print which column we're analyzing
# Rounding the predictions because we're approaching a problem as a regression task
# despite we have 7 output nodes. This way, regression outputs, which would be continuous,
# are seen as classes instead.
estimates = [round(float(x)) for x in y_pred[:, i]]
estimates = transform_results(estimates)
# print(np.unique(estimates)) # Uncomment to see the unique classes in that column
print(classification_report(y_test[:, i], estimates), "\n------") # Print classification report
# Append all the scores to our previously initialized list
acc.append(accuracy_score(y_test[:, i], estimates))
pre.append(precision_score(y_test[:, i], estimates, average='weighted'))
recall.append(recall_score(y_test[:, i], estimates, average='weighted'))
f1.append(f1_score(y_test[:, i], estimates, average='weighted'))
rms.append(np.sqrt(mean_squared_error(y_test[:, i], estimates)))
aucscore.append(score_auc(y_test[:, i], estimates))
mae.append(mean_absolute_error(y_test[:, i], estimates))
Column supportive
precision recall f1-score support
2.0 0.00 0.00 0.00 544
3.0 1.00 0.01 0.02 576
4.0 0.18 1.00 0.31 594
5.0 0.67 0.00 0.01 571
6.0 0.84 0.06 0.11 541
7.0 0.90 0.18 0.29 534
accuracy 0.22 3360
macro avg 0.60 0.21 0.12 3360
weighted avg 0.60 0.22 0.12 3360
------
Column easy
precision recall f1-score support
2.0 0.00 0.00 0.00 544
3.0 1.00 0.01 0.02 575
4.0 0.18 1.00 0.31 589
5.0 0.67 0.00 0.01 573
6.0 0.87 0.06 0.11 541
7.0 0.98 0.19 0.32 538
accuracy 0.22 3360
macro avg 0.62 0.21 0.13 3360
weighted avg 0.61 0.22 0.13 3360
------
Column efficient
precision recall f1-score support
3.0 0.00 0.00 0.00 670
4.0 0.80 0.01 0.01 681
5.0 0.23 1.00 0.37 726
6.0 0.92 0.05 0.10 639
7.0 0.99 0.16 0.28 644
accuracy 0.26 3360
macro avg 0.59 0.24 0.15 3360
weighted avg 0.58 0.26 0.15 3360
------
Column clear
precision recall f1-score support
2.0 0.00 0.00 0.00 544
3.0 1.00 0.01 0.02 578
4.0 0.19 1.00 0.31 595
5.0 0.67 0.00 0.01 567
6.0 0.87 0.06 0.11 540
7.0 0.89 0.17 0.29 536
accuracy 0.22 3360
macro avg 0.60 0.21 0.12 3360
weighted avg 0.60 0.22 0.12 3360
------
Column exciting
precision recall f1-score support
2.0 0.00 0.00 0.00 545
3.0 1.00 0.01 0.02 574
4.0 0.18 1.00 0.30 572
5.0 1.00 0.00 0.01 602
6.0 1.00 0.07 0.13 533
7.0 0.99 0.19 0.32 534
accuracy 0.21 3360
macro avg 0.69 0.21 0.13 3360
weighted avg 0.70 0.21 0.13 3360
------
Column interesting
precision recall f1-score support
2.0 0.00 0.00 0.00 545
3.0 1.00 0.01 0.02 574
4.0 0.18 1.00 0.31 578
5.0 1.00 0.01 0.01 592
6.0 1.00 0.07 0.13 537
7.0 0.99 0.19 0.32 534
accuracy 0.22 3360
macro avg 0.70 0.21 0.13 3360
weighted avg 0.70 0.22 0.13 3360
------
Column inventive
precision recall f1-score support
1.0 0.00 0.00 0.00 545
2.0 1.00 0.01 0.02 576
3.0 0.18 1.00 0.31 589
4.0 1.00 0.01 0.01 586
5.0 1.00 0.07 0.13 533
6.0 0.99 0.19 0.32 531
accuracy 0.22 3360
macro avg 0.70 0.21 0.13 3360
weighted avg 0.69 0.22 0.13 3360
------
Column leading edge
precision recall f1-score support
1.0 0.00 0.00 0.00 544
2.0 1.00 0.01 0.02 576
3.0 0.18 1.00 0.31 582
4.0 1.00 0.01 0.01 589
5.0 1.00 0.07 0.13 536
6.0 1.00 0.20 0.33 533
accuracy 0.22 3360
macro avg 0.70 0.21 0.13 3360
weighted avg 0.70 0.22 0.13 3360
------
Here we display the average of each evaluation metric by calling np.mean() on each array -- remember, each array is comprised of multiple evaluation metrics; one for each column.
print("Average Accuracy : {}".format(np.mean(acc)))
print("Average Precision : {}".format(np.mean(pre)))
print("Average Recall : {}".format(np.mean(recal)))
print("Average F1-score : {}".format(np.mean(f1)))
print("Average RMSE score : {}".format(np.mean(rms)))
print("Average AUC score : {}".format(np.mean(aucscore)))
metric_df['RBFN'] = [np.mean(acc),
np.mean(pre), np.mean(recal),
np.mean(f1), np.mean(rms), np.mean(aucscore),
np.mean(mae)]
Average Accuracy : 0.2220610119047619 Average Precision : 0.6458623685484866 Average Recall : nan Average F1-score : 0.13133872689151005 Average RMSE score : 1.61898293961555 Average AUC score : 0.5259616095945946
LSTM_y_pred = model.predict(X_test)
LSTM_y_pred_flattened = LSTM_y_pred.flatten(order='F')
LSTM_y_classified = [round(float(x)) for x in LSTM_y_pred_flattened]
print("First ten examples of the test set:")
print("Truth labels:", y_test[:10])
print("Predicted labels:", LSTM_y_classified[:10])
print("Overall performance on all columns:\n",
classification_report(y_test.flatten(order='F'),
LSTM_y_classified))
First ten examples of the test set:
Truth labels: [[3. 3. 4. 3. 3. 3. 2. 2.]
[2. 2. 3. 2. 2. 2. 1. 1.]
[3. 3. 3. 3. 3. 3. 2. 2.]
[2. 2. 3. 2. 2. 2. 1. 1.]
[5. 5. 5. 5. 5. 5. 4. 4.]
[4. 4. 4. 4. 4. 4. 3. 3.]
[4. 4. 5. 4. 4. 4. 3. 3.]
[4. 4. 4. 4. 4. 4. 3. 3.]
[4. 4. 5. 4. 4. 4. 3. 3.]
[7. 7. 7. 6. 3. 3. 1. 1.]]
Predicted labels: [4, 4, 4, 4, 4, 4, 4, 4, 4, 4]
Overall performance on all columns:
precision recall f1-score support
1.0 0.00 0.00 0.00 1089
2.0 1.00 0.00 0.01 3874
3.0 0.19 0.25 0.21 4718
4.0 0.18 0.61 0.28 4784
5.0 0.25 0.17 0.20 4700
6.0 0.95 0.09 0.17 4395
7.0 0.96 0.18 0.30 3320
accuracy 0.22 26880
macro avg 0.50 0.19 0.17 26880
weighted avg 0.53 0.22 0.19 26880
The multinomial logistic regression algorithm, an extension to the logistic regression model, involves changing the loss function to cross-entropy loss and predict probability distribution to a multinomial probability distribution to natively support multi-class classification problems.
from sklearn.linear_model import LogisticRegression
acc = []
pre = []
mae = []
recal = []
f1 = []
rms = []
aucscore = []
fpo = dict()
tpo = dict()
ffpo = []
ttpo = []
# Make an instance of LogisticRegression model.
clf = LogisticRegression(multi_class='multinomial', solver='lbfgs')
# Invoke the function from 3.5.1.
plot_all_result_model(clf, X_test, y_train, y_test,
ffpo, ttpo)
For column: supportive
precision recall f1-score support
2.0 0.66 0.86 0.75 544
3.0 0.27 0.20 0.23 576
4.0 0.33 0.09 0.15 594
5.0 0.35 0.19 0.24 571
6.0 0.24 0.42 0.31 541
7.0 0.37 0.56 0.44 534
accuracy 0.38 3360
macro avg 0.37 0.39 0.35 3360
weighted avg 0.37 0.38 0.35 3360
Confusion Matrix:
[[467 76 0 0 1 0]
[118 118 21 35 158 126]
[ 51 71 56 81 191 144]
[ 26 44 50 106 241 104]
[ 14 52 36 70 226 143]
[ 31 69 7 8 120 299]]
Accuracy Score : 0.37857142857142856
Precision Score : 0.3693384539965513
Recall Score : 0.37857142857142856
F1 Score : 0.3480137702485682
For column: easy
precision recall f1-score support
2.0 0.67 0.86 0.75 544
3.0 0.30 0.23 0.26 575
4.0 0.35 0.08 0.14 589
5.0 0.40 0.21 0.27 573
6.0 0.27 0.44 0.34 541
7.0 0.44 0.73 0.55 538
accuracy 0.42 3360
macro avg 0.41 0.43 0.38 3360
weighted avg 0.40 0.42 0.38 3360
Confusion Matrix:
[[467 76 0 0 1 0]
[118 131 9 42 150 125]
[ 43 63 50 65 218 150]
[ 21 43 45 119 237 108]
[ 13 60 36 69 240 123]
[ 33 63 1 2 45 394]]
Accuracy Score : 0.41696428571428573
Precision Score : 0.40416663961206284
Recall Score : 0.41696428571428573
F1 Score : 0.3787764558963471
For column: efficient
precision recall f1-score support
3.0 0.62 0.87 0.72 670
4.0 0.39 0.16 0.22 681
5.0 0.41 0.14 0.21 726
6.0 0.33 0.51 0.40 639
7.0 0.43 0.62 0.51 644
accuracy 0.45 3360
macro avg 0.44 0.46 0.41 3360
weighted avg 0.44 0.45 0.41 3360
Confusion Matrix:
[[580 25 1 37 27]
[152 106 43 233 147]
[ 80 47 100 305 194]
[ 25 43 90 325 156]
[ 98 50 9 88 399]]
Accuracy Score : 0.4494047619047619
Precision Score : 0.4373032669112598
Recall Score : 0.4494047619047619
F1 Score : 0.40743398261398067
For column: clear
precision recall f1-score support
2.0 0.66 0.86 0.75 544
3.0 0.26 0.20 0.23 578
4.0 0.34 0.09 0.14 595
5.0 0.35 0.20 0.26 567
6.0 0.25 0.41 0.31 540
7.0 0.39 0.58 0.47 536
accuracy 0.38 3360
macro avg 0.37 0.39 0.36 3360
weighted avg 0.37 0.38 0.35 3360
Confusion Matrix:
[[467 76 0 0 1 0]
[118 118 17 47 150 128]
[ 49 67 53 83 207 136]
[ 26 45 44 115 239 98]
[ 13 58 35 79 223 132]
[ 31 95 5 6 86 313]]
Accuracy Score : 0.3836309523809524
Precision Score : 0.3728049782011341
Recall Score : 0.3836309523809524
F1 Score : 0.35257621531040634
For column: exciting
precision recall f1-score support
2.0 0.67 0.86 0.75 545
3.0 0.32 0.25 0.28 574
4.0 0.29 0.06 0.11 572
5.0 0.43 0.17 0.24 602
6.0 0.28 0.50 0.36 533
7.0 0.44 0.75 0.56 534
accuracy 0.42 3360
macro avg 0.41 0.43 0.38 3360
weighted avg 0.41 0.42 0.38 3360
Confusion Matrix:
[[468 76 0 0 1 0]
[118 143 18 17 155 123]
[ 46 68 37 60 218 143]
[ 26 52 38 100 269 117]
[ 6 50 34 54 268 121]
[ 32 62 1 0 39 400]]
Accuracy Score : 0.42142857142857143
Precision Score : 0.40507756516407917
Recall Score : 0.42142857142857143
F1 Score : 0.3767688112082379
For column: interesting
precision recall f1-score support
2.0 0.67 0.86 0.75 545
3.0 0.31 0.23 0.26 574
4.0 0.27 0.06 0.10 578
5.0 0.36 0.17 0.23 592
6.0 0.28 0.49 0.36 537
7.0 0.44 0.75 0.56 534
accuracy 0.42 3360
macro avg 0.39 0.43 0.38 3360
weighted avg 0.39 0.42 0.37 3360
Confusion Matrix:
[[468 76 0 0 1 0]
[119 132 17 39 144 123]
[ 48 57 35 78 218 142]
[ 25 54 43 102 257 111]
[ 6 49 35 62 262 123]
[ 32 62 1 1 38 400]]
Accuracy Score : 0.41636904761904764
Precision Score : 0.387113349901577
Recall Score : 0.41636904761904764
F1 Score : 0.3713736445061812
For column: inventive
precision recall f1-score support
1.0 0.67 0.86 0.75 545
2.0 0.30 0.23 0.26 576
3.0 0.29 0.06 0.11 589
4.0 0.39 0.17 0.24 586
5.0 0.28 0.49 0.36 533
6.0 0.44 0.76 0.56 531
accuracy 0.42 3360
macro avg 0.40 0.43 0.38 3360
weighted avg 0.39 0.42 0.37 3360
Confusion Matrix:
[[468 76 0 0 1 0]
[119 133 18 27 152 127]
[ 53 63 38 79 215 141]
[ 20 51 43 102 260 110]
[ 6 57 33 55 259 123]
[ 32 61 0 0 37 401]]
Accuracy Score : 0.41696428571428573
Precision Score : 0.3932819202031307
Recall Score : 0.41696428571428573
F1 Score : 0.3722076511977062
For column: leading edge
precision recall f1-score support
1.0 0.67 0.86 0.76 544
2.0 0.30 0.23 0.26 576
3.0 0.30 0.06 0.10 582
4.0 0.40 0.19 0.25 589
5.0 0.28 0.50 0.36 536
6.0 0.42 0.68 0.52 533
accuracy 0.41 3360
macro avg 0.40 0.42 0.38 3360
weighted avg 0.39 0.41 0.37 3360
Confusion Matrix:
[[468 76 0 0 0 0]
[120 133 10 39 148 126]
[ 46 59 37 75 222 143]
[ 23 53 44 109 252 108]
[ 6 58 32 52 270 118]
[ 32 62 0 0 77 362]]
Accuracy Score : 0.41041666666666665
Precision Score : 0.393766695870301
Recall Score : 0.41041666666666665
F1 Score : 0.36942274286728227
Here we plot the average ROC for each class, found with Multinomial Logistic Regression.
plot_multiclass_roc_optim(clf, X_train, y_train, X_test, y_test)
print("Average Accuracy : {}".format(np.mean(acc)))
print("Average Precision : {}".format(np.mean(pre)))
print("Average Recall : {}".format(np.mean(recal)))
print("Average F1-score : {}".format(np.mean(f1)))
print("Average RMSE score : {}".format(np.mean(rms)))
print("Average AUC score : {}".format(np.mean(aucscore)))
metric_df['LogisticModel'] = [np.mean(acc),
np.mean(pre), np.mean(recal), np.mean(f1),
np.mean(rms), np.mean(aucscore), np.mean(mae)]
Average Accuracy : 0.41171875 Average Precision : 0.395356608732512 Average Recall : 0.41171875 Average F1-score : 0.37207165923108876 Average RMSE score : 1.7162203459369199 Average AUC score : 0.6466900010997682
Long Short Term Memory networks – usually just called “LSTMs” – are a special kind of RNN, capable of learning long-term dependencies. They were introduced by Hochreiter & Schmidhuber (1997), and were refined and popularized by many people in following work.1 They work tremendously well on a large variety of problems, and are now widely used.
LSTMs are explicitly designed to avoid the long-term dependency problem.
# Callbacks
es = tf.keras.callbacks.EarlyStopping(
monitor='val_loss', mode='min', verbose=1, patience=7)
checkpoint = tf.keras.callbacks.ModelCheckpoint(
"model_checkpoints", monitor="val_loss",
verbose=1, save_best_only=True, mode="min")
# Optimizer
opt = tf.keras.optimizers.Adam(learning_rate=0.001)
LSTM_model = tf.keras.Sequential([
tf.keras.layers.Embedding(4000, 128),
tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(128)),
# tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(32)),
# use ReLU in place of tanh function since they are
# very good alternatives of each other.
tf.keras.layers.Dense(64, activation='relu',
# Prevent overfitting
kernel_regularizer=tf.keras.regularizers.L2(
l2=0.01),
kernel_initializer=tf.keras.initializers.RandomNormal(
seed=42)),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(8, activation='linear')])
LSTM_model.summary()
LSTM_model.compile(loss=tf.keras.losses.MeanSquaredError(),
optimizer=opt, metrics=['mae'])
Model: "sequential_2"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
embedding_1 (Embedding) (None, None, 128) 512000
bidirectional_1 (Bidirectio (None, 256) 263168
nal)
dense_11 (Dense) (None, 64) 16448
dropout_6 (Dropout) (None, 64) 0
dense_12 (Dense) (None, 8) 520
=================================================================
Total params: 792,136
Trainable params: 792,136
Non-trainable params: 0
_________________________________________________________________
history = LSTM_model.fit(X_train, y_train, epochs=100,
batch_size=512, validation_split=0.2,
callbacks=[es],
verbose=1)
Epoch 1/100 21/21 [==============================] - 5s 84ms/step - loss: 13.3983 - mae: 3.0659 - val_loss: 3.3809 - val_mae: 1.4924 Epoch 2/100 21/21 [==============================] - 1s 30ms/step - loss: 4.1215 - mae: 1.6364 - val_loss: 2.9511 - val_mae: 1.4268 Epoch 3/100 21/21 [==============================] - 1s 31ms/step - loss: 3.6045 - mae: 1.5431 - val_loss: 2.6733 - val_mae: 1.3674 Epoch 4/100 21/21 [==============================] - 1s 31ms/step - loss: 2.8875 - mae: 1.3634 - val_loss: 1.7285 - val_mae: 1.0212 Epoch 5/100 21/21 [==============================] - 1s 30ms/step - loss: 2.1868 - mae: 1.1153 - val_loss: 1.4085 - val_mae: 0.8812 Epoch 6/100 21/21 [==============================] - 1s 30ms/step - loss: 1.7798 - mae: 0.9732 - val_loss: 0.9341 - val_mae: 0.6629 Epoch 7/100 21/21 [==============================] - 1s 30ms/step - loss: 1.2262 - mae: 0.7721 - val_loss: 0.6104 - val_mae: 0.4724 Epoch 8/100 21/21 [==============================] - 1s 31ms/step - loss: 0.9784 - mae: 0.6635 - val_loss: 0.4840 - val_mae: 0.3598 Epoch 9/100 21/21 [==============================] - 1s 31ms/step - loss: 0.8697 - mae: 0.6117 - val_loss: 0.5069 - val_mae: 0.4055 Epoch 10/100 21/21 [==============================] - 1s 31ms/step - loss: 0.7855 - mae: 0.5744 - val_loss: 0.4231 - val_mae: 0.3281 Epoch 11/100 21/21 [==============================] - 1s 30ms/step - loss: 0.7415 - mae: 0.5551 - val_loss: 0.3761 - val_mae: 0.2860 Epoch 12/100 21/21 [==============================] - 1s 31ms/step - loss: 0.7082 - mae: 0.5419 - val_loss: 0.3798 - val_mae: 0.3062 Epoch 13/100 21/21 [==============================] - 1s 30ms/step - loss: 0.6703 - mae: 0.5252 - val_loss: 0.3508 - val_mae: 0.2730 Epoch 14/100 21/21 [==============================] - 1s 31ms/step - loss: 0.6348 - mae: 0.5112 - val_loss: 0.3353 - val_mae: 0.2858 Epoch 15/100 21/21 [==============================] - 1s 31ms/step - loss: 0.6188 - mae: 0.5063 - val_loss: 0.3332 - val_mae: 0.2673 Epoch 16/100 21/21 [==============================] - 1s 31ms/step - loss: 0.6044 - mae: 0.5005 - val_loss: 0.3132 - val_mae: 0.2713 Epoch 17/100 21/21 [==============================] - 1s 31ms/step - loss: 0.5861 - mae: 0.4932 - val_loss: 0.3007 - val_mae: 0.2658 Epoch 18/100 21/21 [==============================] - 1s 30ms/step - loss: 0.5628 - mae: 0.4838 - val_loss: 0.3101 - val_mae: 0.2739 Epoch 19/100 21/21 [==============================] - 1s 30ms/step - loss: 0.5543 - mae: 0.4810 - val_loss: 0.2847 - val_mae: 0.2481 Epoch 20/100 21/21 [==============================] - 1s 31ms/step - loss: 0.5386 - mae: 0.4757 - val_loss: 0.3028 - val_mae: 0.2811 Epoch 21/100 21/21 [==============================] - 1s 31ms/step - loss: 0.5228 - mae: 0.4700 - val_loss: 0.2842 - val_mae: 0.2651 Epoch 22/100 21/21 [==============================] - 1s 30ms/step - loss: 0.5034 - mae: 0.4609 - val_loss: 0.2651 - val_mae: 0.2388 Epoch 23/100 21/21 [==============================] - 1s 30ms/step - loss: 0.4990 - mae: 0.4615 - val_loss: 0.2554 - val_mae: 0.2233 Epoch 24/100 21/21 [==============================] - 1s 32ms/step - loss: 0.4844 - mae: 0.4539 - val_loss: 0.2515 - val_mae: 0.2335 Epoch 25/100 21/21 [==============================] - 1s 30ms/step - loss: 0.4749 - mae: 0.4494 - val_loss: 0.2458 - val_mae: 0.2237 Epoch 26/100 21/21 [==============================] - 1s 30ms/step - loss: 0.4721 - mae: 0.4519 - val_loss: 0.2401 - val_mae: 0.2410 Epoch 27/100 21/21 [==============================] - 1s 30ms/step - loss: 0.4605 - mae: 0.4465 - val_loss: 0.2354 - val_mae: 0.2201 Epoch 28/100 21/21 [==============================] - 1s 30ms/step - loss: 0.4531 - mae: 0.4443 - val_loss: 0.2467 - val_mae: 0.2414 Epoch 29/100 21/21 [==============================] - 1s 30ms/step - loss: 0.4417 - mae: 0.4394 - val_loss: 0.2294 - val_mae: 0.2206 Epoch 30/100 21/21 [==============================] - 1s 30ms/step - loss: 0.4315 - mae: 0.4347 - val_loss: 0.2219 - val_mae: 0.2291 Epoch 31/100 21/21 [==============================] - 1s 30ms/step - loss: 0.4307 - mae: 0.4363 - val_loss: 0.2525 - val_mae: 0.2800 Epoch 32/100 21/21 [==============================] - 1s 31ms/step - loss: 0.4172 - mae: 0.4286 - val_loss: 0.2219 - val_mae: 0.2303 Epoch 33/100 21/21 [==============================] - 1s 30ms/step - loss: 0.4141 - mae: 0.4293 - val_loss: 0.2174 - val_mae: 0.2251 Epoch 34/100 21/21 [==============================] - 1s 31ms/step - loss: 0.4135 - mae: 0.4294 - val_loss: 0.2237 - val_mae: 0.2401 Epoch 35/100 21/21 [==============================] - 1s 30ms/step - loss: 0.3943 - mae: 0.4188 - val_loss: 0.2069 - val_mae: 0.2190 Epoch 36/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3889 - mae: 0.4156 - val_loss: 0.2023 - val_mae: 0.2217 Epoch 37/100 21/21 [==============================] - 1s 30ms/step - loss: 0.3863 - mae: 0.4182 - val_loss: 0.2028 - val_mae: 0.2237 Epoch 38/100 21/21 [==============================] - 1s 30ms/step - loss: 0.3821 - mae: 0.4132 - val_loss: 0.2085 - val_mae: 0.2414 Epoch 39/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3770 - mae: 0.4118 - val_loss: 0.1990 - val_mae: 0.2156 Epoch 40/100 21/21 [==============================] - 1s 30ms/step - loss: 0.3706 - mae: 0.4078 - val_loss: 0.1951 - val_mae: 0.2135 Epoch 41/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3689 - mae: 0.4077 - val_loss: 0.1890 - val_mae: 0.2041 Epoch 42/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3619 - mae: 0.4054 - val_loss: 0.2021 - val_mae: 0.2288 Epoch 43/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3615 - mae: 0.4054 - val_loss: 0.2001 - val_mae: 0.2327 Epoch 44/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3531 - mae: 0.4003 - val_loss: 0.1840 - val_mae: 0.2115 Epoch 45/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3558 - mae: 0.4043 - val_loss: 0.2013 - val_mae: 0.2406 Epoch 46/100 21/21 [==============================] - 1s 30ms/step - loss: 0.3546 - mae: 0.4017 - val_loss: 0.1888 - val_mae: 0.2098 Epoch 47/100 21/21 [==============================] - 1s 30ms/step - loss: 0.3480 - mae: 0.3997 - val_loss: 0.1851 - val_mae: 0.2240 Epoch 48/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3448 - mae: 0.3975 - val_loss: 0.1802 - val_mae: 0.2066 Epoch 49/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3403 - mae: 0.3977 - val_loss: 0.1779 - val_mae: 0.2122 Epoch 50/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3453 - mae: 0.4003 - val_loss: 0.1791 - val_mae: 0.2129 Epoch 51/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3408 - mae: 0.3972 - val_loss: 0.1800 - val_mae: 0.2230 Epoch 52/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3400 - mae: 0.3991 - val_loss: 0.1923 - val_mae: 0.2441 Epoch 53/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3444 - mae: 0.3996 - val_loss: 0.1727 - val_mae: 0.2099 Epoch 54/100 21/21 [==============================] - 1s 30ms/step - loss: 0.3227 - mae: 0.3859 - val_loss: 0.1724 - val_mae: 0.2016 Epoch 55/100 21/21 [==============================] - 1s 30ms/step - loss: 0.3275 - mae: 0.3904 - val_loss: 0.1837 - val_mae: 0.2247 Epoch 56/100 21/21 [==============================] - 1s 30ms/step - loss: 0.3324 - mae: 0.3953 - val_loss: 0.1760 - val_mae: 0.2225 Epoch 57/100 21/21 [==============================] - 1s 30ms/step - loss: 0.3295 - mae: 0.3926 - val_loss: 0.1797 - val_mae: 0.2356 Epoch 58/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3282 - mae: 0.3908 - val_loss: 0.1792 - val_mae: 0.2265 Epoch 59/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3189 - mae: 0.3838 - val_loss: 0.1690 - val_mae: 0.2113 Epoch 60/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3225 - mae: 0.3884 - val_loss: 0.1825 - val_mae: 0.2364 Epoch 61/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3129 - mae: 0.3808 - val_loss: 0.1666 - val_mae: 0.1976 Epoch 62/100 21/21 [==============================] - 1s 30ms/step - loss: 0.3188 - mae: 0.3859 - val_loss: 0.1901 - val_mae: 0.2520 Epoch 63/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3256 - mae: 0.3914 - val_loss: 0.1701 - val_mae: 0.2136 Epoch 64/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3104 - mae: 0.3793 - val_loss: 0.1761 - val_mae: 0.2355 Epoch 65/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3100 - mae: 0.3778 - val_loss: 0.1694 - val_mae: 0.2250 Epoch 66/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3078 - mae: 0.3776 - val_loss: 0.1666 - val_mae: 0.2082 Epoch 67/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3025 - mae: 0.3755 - val_loss: 0.1693 - val_mae: 0.2126 Epoch 68/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3138 - mae: 0.3835 - val_loss: 0.1642 - val_mae: 0.2048 Epoch 69/100 21/21 [==============================] - 1s 30ms/step - loss: 0.3176 - mae: 0.3858 - val_loss: 0.1636 - val_mae: 0.1962 Epoch 70/100 21/21 [==============================] - 1s 30ms/step - loss: 0.3035 - mae: 0.3729 - val_loss: 0.1627 - val_mae: 0.2009 Epoch 71/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3058 - mae: 0.3755 - val_loss: 0.1817 - val_mae: 0.2401 Epoch 72/100 21/21 [==============================] - 1s 30ms/step - loss: 0.3066 - mae: 0.3767 - val_loss: 0.1764 - val_mae: 0.2307 Epoch 73/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3066 - mae: 0.3787 - val_loss: 0.1713 - val_mae: 0.2108 Epoch 74/100 21/21 [==============================] - 1s 30ms/step - loss: 0.3110 - mae: 0.3813 - val_loss: 0.1734 - val_mae: 0.2301 Epoch 75/100 21/21 [==============================] - 1s 31ms/step - loss: 0.2990 - mae: 0.3729 - val_loss: 0.1637 - val_mae: 0.2274 Epoch 76/100 21/21 [==============================] - 1s 30ms/step - loss: 0.2967 - mae: 0.3716 - val_loss: 0.1604 - val_mae: 0.2053 Epoch 77/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3036 - mae: 0.3743 - val_loss: 0.1639 - val_mae: 0.2111 Epoch 78/100 21/21 [==============================] - 1s 30ms/step - loss: 0.3004 - mae: 0.3738 - val_loss: 0.1613 - val_mae: 0.2047 Epoch 79/100 21/21 [==============================] - 1s 30ms/step - loss: 0.2984 - mae: 0.3726 - val_loss: 0.1630 - val_mae: 0.1896 Epoch 80/100 21/21 [==============================] - 1s 30ms/step - loss: 0.2974 - mae: 0.3715 - val_loss: 0.1635 - val_mae: 0.2029 Epoch 81/100 21/21 [==============================] - 1s 30ms/step - loss: 0.3066 - mae: 0.3796 - val_loss: 0.1613 - val_mae: 0.2206 Epoch 82/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3079 - mae: 0.3811 - val_loss: 0.1662 - val_mae: 0.2112 Epoch 83/100 21/21 [==============================] - 1s 31ms/step - loss: 0.3053 - mae: 0.3792 - val_loss: 0.1606 - val_mae: 0.1982 Epoch 00083: early stopping
plot_graphs(history, "mae")
plot_graphs(history, "loss")
Here we evaluate on y_test, which we made sure earlier comes from a similar distribution within the training set itself.
scores = LSTM_model.evaluate(
X_test, y_test, batch_size=1, verbose=1,
sample_weight=None, steps=None,
callbacks=None, max_queue_size=10,
workers=4, use_multiprocessing=False,
return_dict=False
)
print("------------------ EVALUATION FINISHED! ------------------".center(115))
for i in range(len(scores)):
print("%s: %.2f%%" % (model.metrics_names[i], scores[i]*100))
3360/3360 [==============================] - 18s 5ms/step - loss: 0.1546 - mae: 0.1930
------------------ EVALUATION FINISHED! ------------------
loss: 15.46%
accuracy: 19.30%
Like before, we can initialize some empty list to keep track of the evaluation metrics we're going to use.
acc = []
pre = []
recal = []
f1 = []
mae = []
rms = []
aucscore = []
y_pred = LSTM_model.predict(X_test)
for i in range(8): # Loop over columns
print("Column", df.columns[1:][i]) # Print which column we're analyzing
# Rounding the predictions because we're approaching a problem as a regression task
# despite we have 7 output nodes. This way, regression outputs, which would be continuous,
# are seen as classes instead.
estimates = [round(float(x)) for x in y_pred[:, i]]
estimates = transform_results(estimates)
# print(np.unique(estimates)) # Uncomment to see the unique classes in that column
print(classification_report(y_test[:, i], estimates), "\n------") # Print classification report
# Append all the scores to our previously initialized list
acc.append(accuracy_score(y_test[:, i], estimates))
pre.append(precision_score(y_test[:, i], estimates, average='weighted'))
recall.append(recall_score(y_test[:, i], estimates, average='weighted'))
f1.append(f1_score(y_test[:, i], estimates, average='weighted'))
rms.append(np.sqrt(mean_squared_error(y_test[:, i], estimates)))
aucscore.append(score_auc(y_test[:, i], estimates))
mae.append(mean_absolute_error(y_test[:, i], estimates))
Column supportive
precision recall f1-score support
2.0 1.00 1.00 1.00 544
3.0 0.98 0.98 0.98 576
4.0 0.88 0.83 0.85 594
5.0 0.72 0.77 0.75 571
6.0 0.75 0.75 0.75 541
7.0 0.91 0.90 0.91 534
accuracy 0.87 3360
macro avg 0.87 0.87 0.87 3360
weighted avg 0.87 0.87 0.87 3360
------
Column easy
precision recall f1-score support
2.0 1.00 1.00 1.00 544
3.0 0.98 0.98 0.98 575
4.0 0.89 0.87 0.88 589
5.0 0.78 0.80 0.79 573
6.0 0.82 0.82 0.82 541
7.0 0.98 0.97 0.97 538
accuracy 0.91 3360
macro avg 0.91 0.91 0.91 3360
weighted avg 0.91 0.91 0.91 3360
------
Column efficient
precision recall f1-score support
3.0 1.00 0.81 0.89 670
4.0 0.75 0.66 0.70 681
5.0 0.60 0.87 0.71 726
6.0 0.73 0.74 0.74 639
7.0 0.99 0.82 0.90 644
accuracy 0.78 3360
macro avg 0.82 0.78 0.79 3360
weighted avg 0.81 0.78 0.79 3360
------
Column clear
precision recall f1-score support
2.0 1.00 1.00 1.00 544
3.0 0.98 0.98 0.98 578
4.0 0.88 0.83 0.86 595
5.0 0.72 0.77 0.74 567
6.0 0.76 0.76 0.76 540
7.0 0.91 0.91 0.91 536
accuracy 0.87 3360
macro avg 0.87 0.87 0.87 3360
weighted avg 0.88 0.87 0.87 3360
------
Column exciting
precision recall f1-score support
2.0 1.00 1.00 1.00 545
3.0 0.97 0.97 0.97 574
4.0 0.87 0.87 0.87 572
5.0 0.87 0.85 0.86 602
6.0 0.93 0.94 0.94 533
7.0 0.99 0.99 0.99 534
accuracy 0.94 3360
macro avg 0.94 0.94 0.94 3360
weighted avg 0.94 0.94 0.94 3360
------
Column interesting
precision recall f1-score support
2.0 1.00 1.00 1.00 545
3.0 0.97 0.97 0.97 574
4.0 0.88 0.88 0.88 578
5.0 0.87 0.87 0.87 592
6.0 0.94 0.94 0.94 537
7.0 0.99 0.99 0.99 534
accuracy 0.94 3360
macro avg 0.94 0.94 0.94 3360
weighted avg 0.94 0.94 0.94 3360
------
Column inventive
precision recall f1-score support
1.0 1.00 1.00 1.00 545
2.0 0.99 1.00 0.99 576
3.0 0.95 0.94 0.95 589
4.0 0.93 0.93 0.93 586
5.0 0.97 0.98 0.97 533
6.0 1.00 1.00 1.00 531
accuracy 0.97 3360
macro avg 0.97 0.97 0.97 3360
weighted avg 0.97 0.97 0.97 3360
------
Column leading edge
precision recall f1-score support
1.0 1.00 1.00 1.00 544
2.0 1.00 1.00 1.00 576
3.0 0.98 0.98 0.98 582
4.0 0.99 0.98 0.98 589
5.0 0.99 1.00 0.99 536
6.0 1.00 1.00 1.00 533
accuracy 0.99 3360
macro avg 0.99 0.99 0.99 3360
weighted avg 0.99 0.99 0.99 3360
------
print("Average Accuracy : {}".format(np.mean(acc)))
print("Average Precision : {}".format(np.mean(pre)))
print("Average Recall : {}".format(np.mean(recal)))
print("Average F1 score : {}".format(np.mean(f1)))
print("Average RMSE score : {}".format(np.mean(rms)))
print("Average AUC score : {}".format(np.mean(aucscore)))
metric_df['LSTM'] = [np.mean(acc),
np.mean(pre), np.mean(recal),
np.mean(f1), np.mean(rms),
np.mean(aucscore), np.mean(mae)]
Average Accuracy : 0.9090029761904762 Average Precision : 0.9133378566809738 Average Recall : nan Average F1 score : 0.9099493887898195 Average RMSE score : 0.34899094443932777 Average AUC score : 0.9444715991741285
In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone.
In this project, ensemble algorithms we can try out are as follows:
We've already implemented gradient boosting via XGB on Section 3.5., so we think it's redundant to implement it all over again.
# Import modules
from sklearn.model_selection import cross_val_score
from sklearn.datasets import make_blobs
from sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier, AdaBoostClassifier, BaggingClassifier
Random forests or random decision forests are an ensemble learning method for classification, regression and other tasks that operates by constructing a multitude of decision trees at training time. For classification tasks, the output of the random forest is the class selected by most trees.
acc = []
pre = []
recal = []
f1 = []
mae = []
rms = []
aucscore = []
fpo = dict()
tpo = dict()
ffpo = []
ttpo = []
# Make a RandomForestClassifier object with 100 trees
clf = RandomForestClassifier(n_estimators=100, max_depth=None,
min_samples_split=2, random_state=0)
# Invoke the function from 3.5.1.
plot_all_result_model(clf, X_test, y_train, y_test,
ffpo, ttpo)
For column: supportive
precision recall f1-score support
2.0 1.00 1.00 1.00 544
3.0 0.98 0.98 0.98 576
4.0 0.85 0.88 0.86 594
5.0 0.75 0.71 0.73 571
6.0 0.74 0.75 0.74 541
7.0 0.90 0.90 0.90 534
accuracy 0.87 3360
macro avg 0.87 0.87 0.87 3360
weighted avg 0.87 0.87 0.87 3360
Confusion Matrix:
[[543 1 0 0 0 0]
[ 0 564 12 0 0 0]
[ 0 2 524 61 5 2]
[ 0 1 42 403 123 2]
[ 0 6 31 51 404 49]
[ 1 4 10 21 15 483]]
Accuracy Score : 0.8693452380952381
Precision Score : 0.868442527371826
Recall Score : 0.8693452380952381
F1 Score : 0.8687005235888489
For column: easy
precision recall f1-score support
2.0 1.00 1.00 1.00 544
3.0 0.97 0.98 0.98 575
4.0 0.86 0.89 0.87 589
5.0 0.80 0.75 0.77 573
6.0 0.81 0.82 0.82 541
7.0 0.97 0.97 0.97 538
accuracy 0.90 3360
macro avg 0.90 0.90 0.90 3360
weighted avg 0.90 0.90 0.90 3360
Confusion Matrix:
[[543 1 0 0 0 0]
[ 0 564 11 0 0 0]
[ 0 2 525 58 3 1]
[ 0 3 40 432 97 1]
[ 0 10 33 42 444 12]
[ 1 0 3 11 3 520]]
Accuracy Score : 0.9011904761904762
Precision Score : 0.9006844857683891
Recall Score : 0.9011904761904762
F1 Score : 0.9007764858079118
For column: efficient
precision recall f1-score support
3.0 1.00 0.81 0.89 670
4.0 0.66 0.78 0.71 681
5.0 0.62 0.73 0.67 726
6.0 0.76 0.73 0.75 639
7.0 0.97 0.82 0.89 644
accuracy 0.77 3360
macro avg 0.80 0.77 0.78 3360
weighted avg 0.80 0.77 0.78 3360
Confusion Matrix:
[[543 127 0 0 0]
[ 0 531 147 2 1]
[ 0 124 530 64 8]
[ 0 17 151 466 5]
[ 1 10 23 80 530]]
Accuracy Score : 0.7738095238095238
Precision Score : 0.7981813257809871
Recall Score : 0.7738095238095238
F1 Score : 0.7807742831017614
For column: clear
precision recall f1-score support
2.0 1.00 1.00 1.00 544
3.0 0.98 0.98 0.98 578
4.0 0.86 0.88 0.87 595
5.0 0.75 0.73 0.74 567
6.0 0.75 0.76 0.75 540
7.0 0.91 0.91 0.91 536
accuracy 0.87 3360
macro avg 0.87 0.87 0.87 3360
weighted avg 0.87 0.87 0.87 3360
Confusion Matrix:
[[543 1 0 0 0 0]
[ 0 564 12 2 0 0]
[ 0 2 524 64 4 1]
[ 0 1 30 412 122 2]
[ 1 6 27 53 408 45]
[ 0 4 16 20 9 487]]
Accuracy Score : 0.8744047619047619
Precision Score : 0.873980316093832
Recall Score : 0.8744047619047619
F1 Score : 0.8741451611158875
For column: exciting
precision recall f1-score support
2.0 1.00 1.00 1.00 545
3.0 0.97 0.97 0.97 574
4.0 0.86 0.88 0.87 572
5.0 0.88 0.84 0.86 602
6.0 0.93 0.95 0.94 533
7.0 0.99 0.99 0.99 534
accuracy 0.94 3360
macro avg 0.94 0.94 0.94 3360
weighted avg 0.94 0.94 0.94 3360
Confusion Matrix:
[[543 1 0 1 0 0]
[ 1 558 6 6 3 0]
[ 0 18 506 39 8 1]
[ 0 1 75 505 20 1]
[ 0 0 2 25 504 2]
[ 0 0 0 0 5 529]]
Accuracy Score : 0.9360119047619048
Precision Score : 0.9359483502002582
Recall Score : 0.9360119047619048
F1 Score : 0.935864823705696
For column: interesting
precision recall f1-score support
2.0 1.00 1.00 1.00 545
3.0 0.97 0.97 0.97 574
4.0 0.89 0.90 0.90 578
5.0 0.89 0.87 0.88 592
6.0 0.94 0.94 0.94 537
7.0 0.99 0.99 0.99 534
accuracy 0.94 3360
macro avg 0.95 0.95 0.95 3360
weighted avg 0.94 0.94 0.94 3360
Confusion Matrix:
[[543 1 0 1 0 0]
[ 1 558 8 5 2 0]
[ 0 19 521 29 8 1]
[ 0 0 56 516 19 1]
[ 0 0 1 28 506 2]
[ 0 0 0 0 5 529]]
Accuracy Score : 0.9443452380952381
Precision Score : 0.9442829816991943
Recall Score : 0.9443452380952381
F1 Score : 0.9442842606026127
For column: inventive
precision recall f1-score support
1.0 1.00 1.00 1.00 545
2.0 0.99 1.00 0.99 576
3.0 0.96 0.95 0.96 589
4.0 0.94 0.93 0.94 586
5.0 0.97 0.98 0.97 533
6.0 1.00 1.00 1.00 531
accuracy 0.98 3360
macro avg 0.98 0.98 0.98 3360
weighted avg 0.98 0.98 0.98 3360
Confusion Matrix:
[[544 1 0 0 0 0]
[ 0 574 2 0 0 0]
[ 0 3 562 23 1 0]
[ 0 0 22 547 17 0]
[ 0 0 0 9 522 2]
[ 0 0 0 0 0 531]]
Accuracy Score : 0.9761904761904762
Precision Score : 0.9761145648926977
Recall Score : 0.9761904761904762
F1 Score : 0.9761377390951697
For column: leading edge
precision recall f1-score support
1.0 1.00 1.00 1.00 544
2.0 1.00 1.00 1.00 576
3.0 1.00 1.00 1.00 582
4.0 1.00 0.99 1.00 589
5.0 1.00 1.00 1.00 536
6.0 1.00 1.00 1.00 533
accuracy 1.00 3360
macro avg 1.00 1.00 1.00 3360
weighted avg 1.00 1.00 1.00 3360
Confusion Matrix:
[[544 0 0 0 0 0]
[ 0 576 0 0 0 0]
[ 0 2 580 0 0 0]
[ 0 0 2 586 1 0]
[ 0 0 0 0 536 0]
[ 0 0 0 0 0 533]]
Accuracy Score : 0.9985119047619048
Precision Score : 0.9985145186347079
Recall Score : 0.9985119047619048
F1 Score : 0.9985114194340012
Here we plot the average ROC for each class, found with our Random Forest.
plot_multiclass_roc_optim(clf,X_train, y_train, X_test, y_test)
print("Average Accuracy : {}".format(np.mean(acc)))
print("Average Precision : {}".format(np.mean(pre)))
print("Average Recall : {}".format(np.mean(recal)))
print("Average F1 score : {}".format(np.mean(f1)))
print("Average RMSE score : {}".format(np.mean(rms)))
print("Average AUC score : {}".format(np.mean(aucscore)))
metric_df['RandomForestClassifier'] = [np.mean(acc),
np.mean(pre), np.mean(recal),
np.mean(f1), np.mean(rms),
np.mean(aucscore), np.mean(mae)]
Average Accuracy : 0.9092261904761905 Average Precision : 0.9120186338052365 Average Recall : 0.9092261904761905 Average F1 score : 0.9098993370564862 Average RMSE score : 0.3453807790900732 Average AUC score : 0.9446172394821493
Extremely Randomized (abbreviated Extra) Trees Classifier is a type of ensemble learning technique which aggregates the results of multiple de-correlated decision trees collected in a “forest” to output it's classification result.
# As usual, initialize empty lists to record metrics
acc = []
pre = []
recal = []
mae = []
f1 = []
rms = []
aucscore = []
fpo = dict()
tpo = dict()
ffpo = []
ttpo = []
# Make a ExtraTreesClassifier object with 100 trees
clf = ExtraTreesClassifier(n_estimators=100, max_depth=None,
min_samples_split=2, random_state=0)
# Invoke the function from 3.5.1.
plot_all_result_model(clf, X_test, y_train, y_test,
ffpo, ttpo)
For column: supportive
precision recall f1-score support
2.0 1.00 1.00 1.00 544
3.0 0.98 0.98 0.98 576
4.0 0.83 0.88 0.86 594
5.0 0.75 0.71 0.73 571
6.0 0.75 0.74 0.75 541
7.0 0.90 0.90 0.90 534
accuracy 0.87 3360
macro avg 0.87 0.87 0.87 3360
weighted avg 0.87 0.87 0.87 3360
Confusion Matrix:
[[543 1 0 0 0 0]
[ 0 564 12 0 0 0]
[ 0 2 525 61 4 2]
[ 0 1 51 404 114 1]
[ 0 6 32 51 403 49]
[ 1 4 10 21 15 483]]
Accuracy Score : 0.8696428571428572
Precision Score : 0.8685964615820432
Recall Score : 0.8696428571428572
F1 Score : 0.8688683272105688
For column: easy
precision recall f1-score support
2.0 0.99 1.00 1.00 544
3.0 0.97 0.98 0.98 575
4.0 0.84 0.89 0.87 589
5.0 0.79 0.73 0.76 573
6.0 0.82 0.82 0.82 541
7.0 0.97 0.97 0.97 538
accuracy 0.90 3360
macro avg 0.90 0.90 0.90 3360
weighted avg 0.90 0.90 0.90 3360
Confusion Matrix:
[[543 1 0 0 0 0]
[ 0 564 11 0 0 0]
[ 0 2 526 58 2 1]
[ 2 4 54 421 91 1]
[ 0 10 33 42 444 12]
[ 1 0 3 11 3 520]]
Accuracy Score : 0.8982142857142857
Precision Score : 0.8974614888520085
Recall Score : 0.8982142857142857
F1 Score : 0.8975031779394473
For column: efficient
precision recall f1-score support
3.0 1.00 0.81 0.89 670
4.0 0.64 0.80 0.71 681
5.0 0.62 0.72 0.67 726
6.0 0.78 0.72 0.75 639
7.0 0.98 0.82 0.90 644
accuracy 0.77 3360
macro avg 0.81 0.77 0.78 3360
weighted avg 0.80 0.77 0.78 3360
Confusion Matrix:
[[543 127 0 0 0]
[ 0 545 133 2 1]
[ 0 152 525 45 4]
[ 0 19 158 458 4]
[ 1 10 24 79 530]]
Accuracy Score : 0.7741071428571429
Precision Score : 0.8011917283358213
Recall Score : 0.7741071428571429
F1 Score : 0.7814491679349392
For column: clear
precision recall f1-score support
2.0 1.00 1.00 1.00 544
3.0 0.98 0.98 0.98 578
4.0 0.85 0.88 0.86 595
5.0 0.75 0.72 0.74 567
6.0 0.76 0.76 0.76 540
7.0 0.91 0.91 0.91 536
accuracy 0.87 3360
macro avg 0.87 0.87 0.87 3360
weighted avg 0.87 0.87 0.87 3360
Confusion Matrix:
[[543 1 0 0 0 0]
[ 0 564 12 2 0 0]
[ 0 2 525 64 3 1]
[ 0 1 38 410 117 1]
[ 1 6 28 52 408 45]
[ 0 4 17 19 9 487]]
Accuracy Score : 0.8741071428571429
Precision Score : 0.8734895250118624
Recall Score : 0.8741071428571429
F1 Score : 0.8736920128714736
For column: exciting
precision recall f1-score support
2.0 1.00 1.00 1.00 545
3.0 0.96 0.97 0.97 574
4.0 0.85 0.89 0.87 572
5.0 0.88 0.83 0.86 602
6.0 0.94 0.95 0.94 533
7.0 0.99 0.99 0.99 534
accuracy 0.94 3360
macro avg 0.94 0.94 0.94 3360
weighted avg 0.94 0.94 0.94 3360
Confusion Matrix:
[[543 1 0 1 0 0]
[ 1 558 6 6 3 0]
[ 0 18 510 35 8 1]
[ 0 2 82 500 17 1]
[ 0 0 2 25 504 2]
[ 0 0 0 0 5 529]]
Accuracy Score : 0.9357142857142857
Precision Score : 0.9358577416104952
Recall Score : 0.9357142857142857
F1 Score : 0.9355584160849966
For column: interesting
precision recall f1-score support
2.0 1.00 1.00 1.00 545
3.0 0.96 0.97 0.97 574
4.0 0.87 0.90 0.89 578
5.0 0.89 0.85 0.87 592
6.0 0.94 0.94 0.94 537
7.0 0.99 0.99 0.99 534
accuracy 0.94 3360
macro avg 0.94 0.94 0.94 3360
weighted avg 0.94 0.94 0.94 3360
Confusion Matrix:
[[543 1 0 1 0 0]
[ 1 558 8 5 2 0]
[ 0 19 523 27 8 1]
[ 0 3 67 505 16 1]
[ 0 0 1 28 506 2]
[ 0 0 0 0 5 529]]
Accuracy Score : 0.9416666666666667
Precision Score : 0.9417053827682215
Recall Score : 0.9416666666666667
F1 Score : 0.9415534972512043
For column: inventive
precision recall f1-score support
1.0 1.00 1.00 1.00 545
2.0 0.99 1.00 0.99 576
3.0 0.96 0.95 0.96 589
4.0 0.95 0.94 0.94 586
5.0 0.97 0.98 0.98 533
6.0 1.00 1.00 1.00 531
accuracy 0.98 3360
macro avg 0.98 0.98 0.98 3360
weighted avg 0.98 0.98 0.98 3360
Confusion Matrix:
[[544 1 0 0 0 0]
[ 0 574 2 0 0 0]
[ 0 3 562 23 1 0]
[ 0 0 22 550 14 0]
[ 0 0 0 9 522 2]
[ 0 0 0 0 0 531]]
Accuracy Score : 0.9770833333333333
Precision Score : 0.9770209164282088
Recall Score : 0.9770833333333333
F1 Score : 0.9770457360895483
For column: leading edge
precision recall f1-score support
1.0 1.00 1.00 1.00 544
2.0 1.00 1.00 1.00 576
3.0 1.00 1.00 1.00 582
4.0 1.00 0.99 1.00 589
5.0 1.00 1.00 1.00 536
6.0 1.00 1.00 1.00 533
accuracy 1.00 3360
macro avg 1.00 1.00 1.00 3360
weighted avg 1.00 1.00 1.00 3360
Confusion Matrix:
[[544 0 0 0 0 0]
[ 0 576 0 0 0 0]
[ 0 2 580 0 0 0]
[ 0 0 2 586 1 0]
[ 0 0 0 0 536 0]
[ 0 0 0 0 0 533]]
Accuracy Score : 0.9985119047619048
Precision Score : 0.9985145186347079
Recall Score : 0.9985119047619048
F1 Score : 0.9985114194340012
Here we plot the average ROC for each class, found with our Extra Trees Classifier.
plot_multiclass_roc_optim(clf ,X_train, y_train, X_test, y_test)
print("Average Accuracy : {}".format(np.mean(acc)))
print("Average Precision : {}".format(np.mean(pre)))
print("Average Recall : {}".format(np.mean(recal)))
print("Average F1-score : {}".format(np.mean(f1)))
print("Average RMSE score : {}".format(np.mean(rms)))
print("Average AUC score : {}".format(np.mean(aucscore)))
metric_df['ExtraTreesClassifier'] = [np.mean(acc),np.mean(pre),
np.mean(recal),np.mean(f1),
np.mean(rms),np.mean(aucscore),
np.mean(mae)]
Average Accuracy : 0.9086309523809524 Average Precision : 0.9117297204029211 Average Recall : 0.9086309523809524 Average F1-score : 0.9092727193520225 Average RMSE score : 0.3473821464194616 Average AUC score : 0.9442395493364211
AdaBoost, short for Adaptive Boosting, is a statistical classification meta-algorithm formulated by Yoav Freund and Robert Schapire, who won the 2003 Gödel Prize for their work. It can be used in conjunction with many other types of learning algorithms to improve performance.
acc=[]
pre=[]
recal=[]
f1=[]
mae=[]
rms=[]
aucscore=[]
fpo=dict()
tpo=dict()
ffpo=[]
ttpo=[]
clf = AdaBoostClassifier(n_estimators=500)
plot_all_result_model(clf, X_test, y_train, y_test,
ffpo, ttpo)
For column: supportive
precision recall f1-score support
2.0 0.98 0.86 0.91 544
3.0 0.73 0.16 0.27 576
4.0 0.26 0.19 0.22 594
5.0 0.22 0.75 0.34 571
6.0 0.18 0.09 0.12 541
7.0 0.34 0.05 0.09 534
accuracy 0.35 3360
macro avg 0.45 0.35 0.32 3360
weighted avg 0.45 0.35 0.32 3360
Confusion Matrix:
[[468 0 0 76 0 0]
[ 10 95 84 290 97 0]
[ 0 14 112 396 50 22]
[ 0 13 76 426 40 16]
[ 0 8 65 405 48 15]
[ 1 1 98 373 34 27]]
Accuracy Score : 0.35
Precision Score : 0.44721482901417703
Recall Score : 0.35
F1 Score : 0.32282099970032646
For column: easy
precision recall f1-score support
2.0 1.00 0.67 0.80 544
3.0 0.68 0.21 0.33 575
4.0 0.22 0.09 0.13 589
5.0 0.22 0.84 0.35 573
6.0 0.05 0.03 0.03 541
7.0 0.19 0.06 0.09 538
accuracy 0.32 3360
macro avg 0.39 0.32 0.29 3360
weighted avg 0.39 0.32 0.29 3360
Confusion Matrix:
[[367 0 0 76 101 0]
[ 0 123 59 301 41 51]
[ 0 24 52 452 33 28]
[ 0 15 38 479 24 17]
[ 0 19 13 464 14 31]
[ 1 0 73 374 60 30]]
Accuracy Score : 0.3169642857142857
Precision Score : 0.39346507904740946
Recall Score : 0.3169642857142857
F1 Score : 0.2875628320820745
For column: efficient
precision recall f1-score support
3.0 0.92 0.71 0.80 670
4.0 0.54 0.66 0.59 681
5.0 0.38 0.39 0.38 726
6.0 0.43 0.36 0.39 639
7.0 0.53 0.61 0.57 644
accuracy 0.54 3360
macro avg 0.56 0.54 0.55 3360
weighted avg 0.56 0.54 0.55 3360
Confusion Matrix:
[[477 112 75 2 4]
[ 25 450 94 57 55]
[ 7 145 282 139 153]
[ 7 70 202 227 133]
[ 2 57 92 103 390]]
Accuracy Score : 0.5434523809523809
Precision Score : 0.5582318706694142
Recall Score : 0.5434523809523809
F1 Score : 0.5457730825548951
For column: clear
precision recall f1-score support
2.0 0.98 0.86 0.91 544
3.0 0.40 0.35 0.37 578
4.0 0.20 0.29 0.24 595
5.0 0.23 0.42 0.30 567
6.0 0.20 0.14 0.16 540
7.0 0.43 0.10 0.16 536
accuracy 0.36 3360
macro avg 0.41 0.36 0.36 3360
weighted avg 0.40 0.36 0.36 3360
Confusion Matrix:
[[468 1 0 75 0 0]
[ 10 204 87 191 64 22]
[ 0 92 170 252 67 14]
[ 0 70 188 237 54 18]
[ 1 68 177 203 73 18]
[ 0 78 227 75 102 54]]
Accuracy Score : 0.35892857142857143
Precision Score : 0.40172496057877294
Recall Score : 0.35892857142857143
F1 Score : 0.35625144917228696
For column: exciting
precision recall f1-score support
2.0 0.86 0.48 0.62 545
3.0 0.40 0.47 0.43 574
4.0 0.17 0.13 0.15 572
5.0 0.19 0.19 0.19 602
6.0 0.23 0.44 0.31 533
7.0 0.32 0.19 0.24 534
accuracy 0.32 3360
macro avg 0.36 0.32 0.32 3360
weighted avg 0.36 0.32 0.32 3360
Confusion Matrix:
[[264 0 206 75 0 0]
[ 17 270 37 100 150 0]
[ 14 135 74 120 175 54]
[ 5 107 37 116 243 94]
[ 8 99 10 113 234 69]
[ 0 71 71 96 195 101]]
Accuracy Score : 0.3151785714285714
Precision Score : 0.3568527766529385
Recall Score : 0.3151785714285714
F1 Score : 0.3190874249896289
For column: interesting
precision recall f1-score support
2.0 0.96 0.48 0.64 545
3.0 0.32 0.54 0.40 574
4.0 0.20 0.12 0.15 578
5.0 0.26 0.16 0.20 592
6.0 0.27 0.28 0.27 537
7.0 0.18 0.29 0.22 534
accuracy 0.31 3360
macro avg 0.37 0.31 0.32 3360
weighted avg 0.36 0.31 0.31 3360
Confusion Matrix:
[[263 105 102 75 0 0]
[ 11 312 29 47 78 97]
[ 0 180 69 83 78 168]
[ 0 124 49 97 106 216]
[ 0 91 19 73 149 205]
[ 0 172 71 1 136 154]]
Accuracy Score : 0.3107142857142857
Precision Score : 0.3629958068746841
Recall Score : 0.3107142857142857
F1 Score : 0.31335593610141577
For column: inventive
precision recall f1-score support
1.0 0.99 0.48 0.65 545
2.0 0.45 0.51 0.48 576
3.0 0.19 0.25 0.21 589
4.0 0.21 0.11 0.15 586
5.0 0.23 0.18 0.20 533
6.0 0.17 0.30 0.22 531
accuracy 0.31 3360
macro avg 0.37 0.31 0.32 3360
weighted avg 0.37 0.31 0.32 3360
Confusion Matrix:
[[264 105 101 75 0 0]
[ 0 294 88 44 16 134]
[ 0 127 149 56 77 180]
[ 2 62 161 65 61 235]
[ 0 26 152 30 94 231]
[ 0 35 147 33 157 159]]
Accuracy Score : 0.30505952380952384
Precision Score : 0.37236341060257233
Recall Score : 0.30505952380952384
F1 Score : 0.3170350065424405
For column: leading edge
precision recall f1-score support
1.0 1.00 0.86 0.93 544
2.0 0.59 0.35 0.44 576
3.0 0.24 0.16 0.19 582
4.0 0.26 0.20 0.23 589
5.0 0.22 0.33 0.26 536
6.0 0.15 0.26 0.19 533
accuracy 0.36 3360
macro avg 0.41 0.36 0.37 3360
weighted avg 0.41 0.36 0.37 3360
Confusion Matrix:
[[469 0 0 75 0 0]
[ 0 201 86 17 99 173]
[ 0 56 94 55 168 209]
[ 0 31 62 117 179 200]
[ 0 20 74 73 176 193]
[ 0 35 71 109 181 137]]
Accuracy Score : 0.35535714285714287
Precision Score : 0.4092153245893927
Recall Score : 0.35535714285714287
F1 Score : 0.3701602165060013
Here we plot the average ROC for each class, found with our AdaBoost.
plot_multiclass_roc_optim(clf, X_train, y_train, X_test, y_test)
print("Average Accuracy : {}".format(np.mean(acc)))
print("Average Precision : {}".format(np.mean(pre)))
print("Average Recall : {}".format(np.mean(recal)))
print("Average F1-score : {}".format(np.mean(f1)))
print("Average RMSE score : {}".format(np.mean(rms)))
print("Average AUC score : {}".format(np.mean(aucscore)))
metric_df['AdaBoost'] = [np.mean(acc), np.mean(pre),
np.mean(recal), np.mean(f1),
np.mean(rms), np.mean(aucscore),
np.mean(mae)]
Average Accuracy : 0.35695684523809523 Average Precision : 0.4127580072536702 Average Recall : 0.35695684523809523 Average F1-score : 0.3540058684561337 Average RMSE score : 1.616387818911226 Average AUC score : 0.612222885380761
A Bagging classifier is an ensemble meta-estimator that fits base classifiers each on random subsets of the original dataset and then aggregate their individual predictions (either by voting or by averaging) to form a final prediction. Such a meta-estimator can typically be used as a way to reduce the variance of a black-box estimator (e.g., a decision tree), by introducing randomization into its construction procedure and then making an ensemble out of it.
from sklearn.model_selection import cross_val_score, RepeatedStratifiedKFold
from sklearn.neighbors import KNeighborsClassifier
acc=[]
pre=[]
recal=[]
f1=[]
rms=[]
aucscore=[]
mae=[]
fpo=dict()
tpo=dict()
ffpo=[]
ttpo=[]
clf = BaggingClassifier(KNeighborsClassifier(),
max_samples=0.5, max_features=0.5)
plot_all_result_model(clf, X_test, y_train, y_test,
ffpo, ttpo)
For column: supportive
precision recall f1-score support
2.0 1.00 1.00 1.00 544
3.0 0.96 0.98 0.97 576
4.0 0.82 0.86 0.84 594
5.0 0.74 0.68 0.71 571
6.0 0.73 0.74 0.74 541
7.0 0.90 0.90 0.90 534
accuracy 0.86 3360
macro avg 0.86 0.86 0.86 3360
weighted avg 0.86 0.86 0.86 3360
Confusion Matrix:
[[543 1 0 0 0 0]
[ 0 564 12 0 0 0]
[ 0 6 510 62 15 1]
[ 0 2 59 389 117 4]
[ 0 7 32 53 400 49]
[ 1 5 9 21 15 483]]
Accuracy Score : 0.8598214285714286
Precision Score : 0.858441307890437
Recall Score : 0.8598214285714286
F1 Score : 0.8588250906935916
For column: easy
precision recall f1-score support
2.0 1.00 1.00 1.00 544
3.0 0.96 0.98 0.97 575
4.0 0.82 0.87 0.84 589
5.0 0.77 0.71 0.74 573
6.0 0.81 0.82 0.82 541
7.0 0.97 0.97 0.97 538
accuracy 0.89 3360
macro avg 0.89 0.89 0.89 3360
weighted avg 0.89 0.89 0.89 3360
Confusion Matrix:
[[543 1 0 0 0 0]
[ 0 564 11 0 0 0]
[ 0 3 510 70 6 0]
[ 0 6 62 404 98 3]
[ 0 11 31 41 446 12]
[ 1 0 6 8 3 520]]
Accuracy Score : 0.8889880952380952
Precision Score : 0.888011925541751
Recall Score : 0.8889880952380952
F1 Score : 0.888111129051476
For column: efficient
precision recall f1-score support
3.0 1.00 0.81 0.89 670
4.0 0.66 0.76 0.71 681
5.0 0.62 0.73 0.67 726
6.0 0.74 0.72 0.73 639
7.0 0.97 0.82 0.89 644
accuracy 0.77 3360
macro avg 0.80 0.77 0.78 3360
weighted avg 0.79 0.77 0.78 3360
Confusion Matrix:
[[543 127 0 0 0]
[ 0 519 160 2 0]
[ 0 109 533 79 5]
[ 0 18 148 463 10]
[ 1 10 22 80 531]]
Accuracy Score : 0.7705357142857143
Precision Score : 0.7943403895734718
Recall Score : 0.7705357142857143
F1 Score : 0.7775213764212374
For column: clear
precision recall f1-score support
2.0 1.00 1.00 1.00 544
3.0 0.97 0.98 0.97 578
4.0 0.80 0.86 0.83 595
5.0 0.72 0.66 0.69 567
6.0 0.74 0.74 0.74 540
7.0 0.91 0.91 0.91 536
accuracy 0.86 3360
macro avg 0.86 0.86 0.86 3360
weighted avg 0.85 0.86 0.86 3360
Confusion Matrix:
[[543 1 0 0 0 0]
[ 0 564 12 2 0 0]
[ 0 4 511 70 9 1]
[ 0 2 64 374 123 4]
[ 1 7 31 58 398 45]
[ 0 5 19 16 9 487]]
Accuracy Score : 0.85625
Precision Score : 0.8547934532860608
Recall Score : 0.85625
F1 Score : 0.8551299733441626
For column: exciting
precision recall f1-score support
2.0 1.00 1.00 1.00 545
3.0 0.95 0.97 0.96 574
4.0 0.85 0.85 0.85 572
5.0 0.86 0.81 0.83 602
6.0 0.91 0.95 0.93 533
7.0 0.99 0.99 0.99 534
accuracy 0.93 3360
macro avg 0.93 0.93 0.93 3360
weighted avg 0.92 0.93 0.93 3360
Confusion Matrix:
[[543 1 0 1 0 0]
[ 1 558 7 5 3 0]
[ 0 20 487 51 13 1]
[ 0 7 77 489 26 3]
[ 0 0 2 25 504 2]
[ 0 0 0 0 5 529]]
Accuracy Score : 0.9255952380952381
Precision Score : 0.9249453553982382
Recall Score : 0.9255952380952381
F1 Score : 0.925107231026216
For column: interesting
precision recall f1-score support
2.0 1.00 1.00 1.00 545
3.0 0.96 0.97 0.97 574
4.0 0.85 0.88 0.87 578
5.0 0.88 0.80 0.84 592
6.0 0.91 0.94 0.93 537
7.0 0.99 0.99 0.99 534
accuracy 0.93 3360
macro avg 0.93 0.93 0.93 3360
weighted avg 0.93 0.93 0.93 3360
Confusion Matrix:
[[543 1 0 0 1 0]
[ 1 558 10 3 2 0]
[ 0 19 511 34 13 1]
[ 0 4 79 473 31 5]
[ 0 0 1 27 507 2]
[ 0 0 0 0 5 529]]
Accuracy Score : 0.9288690476190476
Precision Score : 0.9286632701730821
Recall Score : 0.9288690476190476
F1 Score : 0.9282893240790718
For column: inventive
precision recall f1-score support
1.0 1.00 1.00 1.00 545
2.0 0.98 1.00 0.99 576
3.0 0.93 0.92 0.92 589
4.0 0.92 0.87 0.90 586
5.0 0.93 0.97 0.95 533
6.0 0.98 1.00 0.99 531
accuracy 0.96 3360
macro avg 0.96 0.96 0.96 3360
weighted avg 0.96 0.96 0.96 3360
Confusion Matrix:
[[544 1 0 0 0 0]
[ 0 574 2 0 0 0]
[ 0 7 539 37 6 0]
[ 0 1 38 512 31 4]
[ 0 0 0 9 517 7]
[ 0 0 0 0 0 531]]
Accuracy Score : 0.9574404761904762
Precision Score : 0.9570639125129632
Recall Score : 0.9574404761904762
F1 Score : 0.9570676479778185
For column: leading edge
precision recall f1-score support
1.0 1.00 1.00 1.00 544
2.0 0.99 1.00 0.99 576
3.0 0.97 0.98 0.97 582
4.0 0.98 0.93 0.96 589
5.0 0.97 1.00 0.99 536
6.0 1.00 1.00 1.00 533
accuracy 0.98 3360
macro avg 0.98 0.98 0.98 3360
weighted avg 0.98 0.98 0.98 3360
Confusion Matrix:
[[544 0 0 0 0 0]
[ 0 576 0 0 0 0]
[ 0 2 568 11 1 0]
[ 0 5 18 549 15 2]
[ 0 0 0 0 536 0]
[ 0 0 0 0 0 533]]
Accuracy Score : 0.9839285714285714
Precision Score : 0.9839608680999867
Recall Score : 0.9839285714285714
F1 Score : 0.9837951934578109
Here we plot the average ROC for each class, found with our Bagging Classifier.
plot_multiclass_roc_optim(clf ,X_train, y_train, X_test, y_test)
print("average Accuracy :{}".format(np.mean(acc)))
print("average Precision :{}".format(np.mean(pre)))
print("average Recall :{}".format(np.mean(recal)))
print("average F1 -score :{}".format(np.mean(f1)))
print("average RMSE {}".format(np.mean(rms)))
print("average AUC score {}".format(np.mean(aucscore)))
metric_df['BaggingClassifier'] = [np.mean(acc), np.mean(pre),
np.mean(recal), np.mean(f1),
np.mean(rms), np.mean(aucscore),
np.mean(mae)]
average Accuracy :0.8964285714285715 average Precision :0.8987775603094988 average Recall :0.8964285714285715 average F1 -score :0.8967308707564231 average RMSE 0.388894940005575 average AUC score 0.936900933815113
metric_df
| Neural Network Classifier | XGB | SVM | OrdinalModel | Bidirectional LSTM | RBFN | LogisticModel | LSTM | RandomForestClassifier | ExtraTreesClassifier | AdaBoost | BaggingClassifier | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| metric | ||||||||||||
| Accuracy | 0.893452 | 0.811161 | 0.788653 | 0.788021 | 0.901414 | 0.222061 | 0.411719 | 0.909003 | 0.909226 | 0.908631 | 0.356957 | 0.896429 |
| Precision | 0.897508 | 0.809607 | 0.791142 | 0.784656 | 0.905717 | 0.645862 | 0.395357 | 0.913338 | 0.912019 | 0.911730 | 0.412758 | 0.898778 |
| Recall | 0.893452 | 0.811161 | 0.788653 | 0.788021 | 0.901414 | NaN | 0.411719 | NaN | 0.909226 | 0.908631 | 0.356957 | 0.896429 |
| F1 Score | 0.894045 | 0.807459 | 0.783608 | 0.783826 | 0.902197 | 0.131339 | 0.372072 | 0.909949 | 0.909899 | 0.909273 | 0.354006 | 0.896731 |
| RMSE | 0.383763 | 0.629054 | 0.783906 | 0.642674 | 0.365639 | 1.618983 | 1.716220 | 0.348991 | 0.345381 | 0.347382 | 1.616388 | 0.388895 |
| AUC score | 0.935121 | 0.885471 | 0.871873 | 0.871510 | 0.939893 | 0.525962 | 0.646690 | 0.944472 | 0.944617 | 0.944240 | 0.612223 | 0.936901 |
| MAE | 0.122693 | 0.251749 | 0.318564 | 0.272805 | 0.114174 | 1.320461 | 1.156808 | 0.105990 | 0.107106 | 0.107924 | 1.171131 | 0.122879 |
# metric_df.to_csv("final_result.csv") # Uncomment to save as .csv file